MongoDB profiling: writing CLI for tracing Query Logs in GO using the new driver and go modules

We will write a small tool that will help us with mongo query profiling and tracing. It’s very handy when you want to see all queries currently executed in MongoDB. I use it mostly to find long-running queries, queries that don’t use indexes or tracing a business flow in case of a bug.

Main requirements

  • Print currently executing queries to stdout in raw or human-friendly form
  • Knows to work with a replica set
  • Support for config files (cause we have many environments)

The research

Before we start, let’s talk about MongoDB profiler. All the information we need to be able to write our tool exists on the documentation page:

Let’s code

I will create the project using a relatively new “modules” feature + vendoring.

mkdir -p ~/goprojects/mongotrace
cd ~/goprojects/mongotrace
go mod init github.com/spaiz/mongotrace
main.go

CLI interface

Our tool will support multiple commands: version, info, enable, disable, tail, where the tail command will support multiple options like colored output, indentation and other. Let’s see some of the usage examples:

mongotrace version
mongotrace info --config="./config.local.json"
mongotrace enable --config="./config.local.json"
mongotrace tail --config="./config.local.json"
mongotrace tail --raw --indent=false --config="./config.local.json"
app.go

Config

For the config, we will use a simple structure and simple JSON parsing. The only requirements there are to support multiple database connections:

config.local.json
config.staging.json
config.go

MongoDB Connection

I created a tiny wrapper over the MongoDB connection with a few helper methods. The code is quite straightforward.

Commands

Each command will be implemented as an independent entity that implements the Command interface (kind of Command Pattern):

type Command interface {
Execute() error
}
cmd_report.go

Formatters

I have separate formatter per MongoDB command. Each formatter tries to “reverse engineer” the log back to be a valid MongoDB command. It’s not always possible, cause MongoDB has bugs like this one. Also, the log document may have a different structure in different MongoDB versions. So, it’s not very reliable but sometimes just helpful to see what is going on now in the database.

example for the GROUP query

Publishing

Cause we used the modules feature, the flow of the building and publishing is a little different. Now, the build command will not only build the binary but also will update the go.mod file with the used packages.

go build
go mod vendor
go build -mod vendor
git init 
git add .
git commit -m "initial commit"
git remote add origin git@github.com:spaiz/mongotrace.git
git push -u origin master
git tag v0.0.2
git push --tags

Bonus

I created a tiny Makefile to help me to build and install the binary with supporting binary versioning.

make build
make install

TIP

When you want to see the queries that are triggered by some specific feature in non-local environment, I suggest you use MongoDB’s $comment field. You can pass the request-id (it has many names like correlation-id, trace-id, etc) to be the $comment in the query. Then you will see this value in the logs.

db.mycollection.find({
name: 'Alex',
$comment: 'request-id-fe4a6604-ec0e-40ec-a72c-6f3b899b7d4b'
});

In GO we trust. Software Engineer.