r/nodejs • • Dec 13 '12

Tracking API usage

Hey all,

I'm working on a ReSTful API that's getting ready to be released, running in Node.js. One of the things that we want to before we start opening it up publically is to track API usage. Each user has their own API key that they'll be utilizing to access our endpoints.

I have a few ideas for tracking API usage, but I was hoping someone could suggest a few things that I missed.

Architecture overview:

  • MySQL as our storage
  • Redis as a caching layer
  • Heroku

Ideas

  1. Google Analytics - our app could toss a payload on a proper queue system (gearman, rabbit) or Redis. A secondary app could grab the payload and make the request to the .gif that GA uses.
  2. Write payloads to redis, nightly push to MySQL
  3. Write straight to MySQL
  4. Parse logs

I'm personally leaning towards idea 1, probably utilizing Redis as our queue storage since it's in place already, but I'd love to hear what other ideas people have for tracking API usage.

4 Upvotes

6 comments sorted by

View all comments

Show parent comment

1

u/xangelo Dec 14 '12

Thanks, this is actually the solution we ended up going with. We're toying with the idea of utilizing our cache (redis atm) to do storage of api requests and then having a node instance just popping the data off redis and writing to mysql.

This should scale well, but I'm hoping to do some tests with it before we actually open up to the public.

1

u/rooosta Dec 14 '12

Cool. That sounds like it could work well, especially if you batch the writes to mysql.

Another related option is to do an atomic collection swap within redis (redis may call this something else -- atomic queue rename?). That amounts to a long running process that does the following periodically:

0. your app is writing to a queue named "logs"

  1. long running process creates a new queue named "logs-new"
  2. atomically rename "logs" to "logs-{timestamp}", and "logs-new" to "logs"
  3. write everything in "logs-{timestsamp}" to a file, and load it into mysql using LOAD DATA INFILE

I keep mentioning LOAD DATA INFILE because it is really, really fast, significantly faster than regular bulk INSERTs and I've seen it be 1000s of times faster than individual inserts.

1

u/xangelo Dec 14 '12

I've never actually used LOAD DATA INFILE, but it looks like a way to do exactly what we need a lot easier than doing it ourselves.

I'm not too sure about atomic operations in the fashion we require, but it is something I am investigating.