API Everything

Operation Chatops

Persistence Setup

Setting Up Bot Persistence
See HERE for Initial Bot Setup instructions.

Picking up where I left off with the ChatOps bot configuration. Power outages are a thing, and battery backups in my server rack are not.
So, before going any further with the bot, let’s make sure that at its core it has some built in persistence.
Two things are needed to survive a reboot:

1. The bot’s running python script.
2. The ngrok url hardcoded into the python script. This url changes each time ngrok stops/restarts, thus the bot’s ngrok_url value becomes obsolete upon restart.

Python Script Persistence

The first and easiest thing to do is set up a systemd service in the Ubuntu Server which starts up the bot’s python code each time system starts up.
Log into the server and make /etc/systemd/system/webexbot.service
In that file, add the following, being careful to edit anything within the <carrots>.

[Unit]
Description=<Webex Bot with Webhooks and Ngrok>
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=<username>
ExecStart=<path/to/python3> <path/to/bot.py>

[Install]
WantedBy=multi-user.target

Start the service with sudo systemctl start webexbot
Then enable the service (on restart) with sudo systemctl enable webexbot
Great, now the bot’s python code will run each time the system is restarted, but only AFTER the network services have started. On to the tougher part, circumventing ngrok paywall features!

Ngrok URL Persistence

Ngrok has a nifty low dollar monthly subscription which provides statuc endpoint URLs. I have a finite amount of low dollar monthly subscription shekels, which are currently already allocated to other services. You probably do, too. So, I needed, and have found, a way around this.
First, enable ngrok logs.
Open "~/.config/ngrok/ngrok.yml" and add the line log: /path/to/ngrok.log
Great! If you run and re-run ngrok a few times, you will see that ngrok spits out the new endpoint url at the end of a log line, as url=https://<4charHex>-your-ip-number.ngrok-free.app
That can be found with the following regex: "url=https://[A-Za-z0-9\-]+\.ngrok-free\.app"

Problem: Python reads the file from top to bottom, it will always match on the very first url listed in the log, which after the first run is always outdated. The newest endpoint url will always be at the BOTTOM of the log.
Solution: Python has a file_read_backwards module which…reads the file from bottom to top. So, you can use this to read logs bottom up, instead of top down.

With a little bit of for-loop, a try/except, some regex, and bracket stripping, and we have a solid solution.
Let’s also move ngrok initialization into the python script, and add some logging.
Install some new packages via pip install file_read_backwards, os, re, logging
Add this to your bot code, test it yourself but it’ll work, edit anything within <carrots>:

# new imports at top of code
import os
from file_read_backwards import FileReadBackwards
import re
import logging

# Setup logging
logging.basicConfig(filename='<path/to/>webexbot.log', filemode='w', format='%(asctime)s - %(levelname)s - '
                                                                                      '%(message)s',
                    datefmt='%d-%b-%y %H:%M:%S', level=logging.DEBUG)

# Start Ngrok from within the bot's python code, then wait 5 seconds for logs to show up
logging.info("Starting ngrok (http) on port <ngrok_port>, then sleeping for 5 sec...")
os.system("ngrok http <ngrok_port> &")
os.system("sleep 5")

# open ngrok.log with file_read_backwards, to read oldest log lines first
with FileReadBackwards("<path/to/>ngrok.log", encoding="utf-8") as frb:
    # look line by line up through the log file
    for line in frb:
        # try this & if it fails, then see the 'except'
        try:  
            new_url = str(line)
            logging.info("Looking for most recent string url=...")
            # look for the 'url=https://[gobbledeegook].ngrok-free.app' characters
            url_chunk = re.findall(r"url=https://[A-Za-z0-9\-]+\.ngrok-free\.app", new_url)
            # ignore all the non-matching lines that will print '[]'
            if str(url_chunk) != "[]":
                # for any actual matches, strip out the 'url=' part
                url = re.findall(r"https://[A-Za-z0-9\-]+\.ngrok-free\.app", str(url_chunk))
                # strip out the brackets
                ngrok_logged_url = url[0]
                logging.info("...Found ngrok.log url: " + ngrok_logged_url)
                # immediately EXIT THE LOOP, creating a "first match wins" scenario
                break
        # if the above failed, then do not pass go, QUIT THE SCRIPT!
        except:
            logging.critical("...FAILED to find ngrok.log url via regex.")
            quit()
            
# and finally, replace the old hardcoded bot_url with the dynamically updating logged url variable
bot_url = ngrok_logged_url

# continue on with the rest of the script

And there you have it, the ngrok endpoint url now dynamically changes whenever the python script is re-run.

And the python script will re-run anytime the server restarts.

Also, no longer have to manually start ngrok before running the bot’s python script!

In addition to, you can use that monthly ngrok subscription money for a TryHackMe subscription!

WINNING!

Next up will be adding some test API functionality to the bot via a new command. Then I can tweak that to make it do whatever I need it to do with OSSEC agents and what-have-you.