I’m working on a project that involves aggregating data from many sources (Twitter, Blogs, Yammer, etc) so my team can get visibility and respond to questions being asked about our product. My goal is to suck in all the data from those sources, pop it into a database and then create a Power BI dashboard out of it.
To do so, I created a webhook endpoint that accepts HTTP POST operations so I can have a single endpoint that accepts data and inserts it into the database. The missing part was querying for the data and calling my webhook. Instead of writing all those connectors I would rather use a service that does that for me. There are a bunch of services out there that enable service to service integrations, like Zapier and IFTTT. There’s a good list of them here. I spent a bunch of time with Zapier because they have connectors to most of the services I need to connect with and they support webhooks. IFTTT does those things as well, but I found their product to be too buggy for me to work with, especially the webhook (aka Maker) connector.
Zapier is a polished product with an amazing user experience, but there are a few limitations that I think you should know about. I want to make sure you know that I think very highly of Zapier as a product. The UX is amazing. The error messaging is great. The support team is phenomenal. There are just two important things that prevent me from using it….
It only retrieves new data
Zapier is for “new” data. Any new piece of content that is created in your source connector will be sent to your destination connector. But, if you need old data you have to get that yourself. It doesn’t allow you to go back in time. For my project, I have to get old and new data from the source. I could still use Zapier for new data, but I would have to write a script to get old data to prime it. If I go that far to get old data, then I can just quickly write a script to ping for new data. So, that alone would prevent me from using Zapier.
I asked Zapier if they plan to expose direct access to their connectors so I could use then when creating my code to get old data. They unfortunately said no.
My suggestion to Zapier is to allow for historical data retrieval or give us access to the connectors so we can prime with old data and then continue to use Zaps for new data.
It can get costly
Zapier has “Zaps” and “Tasks”. A Zap is equivalent to a job that connects source and destination data and a Task is equivalent to a POST operation to your destination. They charge per Task, not per run of the Zap. Tasks can add up quickly, especially if you have a lot of connectors that are popular.
I incorrectly assumed that a Task was a “run of the Zap”, but it’s not. A Task is “an execution of your destination connector”. If your destination is a webhook, then each time they call that webhook will count as a Task.
As you can see on their pricing page you have multiple plans to choose from.
In my case, Twitter is only one of 20+ connectors that I need to setup. Even at the Enterprise plan at $400/month or $4,800 a year the Task allocation of 200,000 a month wouldn’t be enough.
**My suggestion to Zapier would be to charge per run of a Zap, not per Task. **
I want to stress that Zapier is a great product – but from my view it has a flawed billing model. It might be perfect for you, if you 1) only need new data 2) are** okay with their pricing model**. Just make sure you understand that a Task is per Tweet, not per run of the Zap.
Hope this helps.
Jon