Top products from r/tensorflow

As is, you probably won't be able to solve this problem. As you said, importing Tensorflow takes a significant amount of time to import, and there's just not going to be a way to speed that up within your current workflow. Of course, you have a lot of options.

First, if you're only using this for inference, you don't need to import all of Tensorflow. I'm not sure of the specifics, but you can most likely trim down your imports to use only the specific functions and classes that you need. You'll likely have to refactor your code and/or prepare you model differently, but it shouldn't be too much effort since light-weight inference is an important aspect of deep learning. As u/MogwaiAllOnYourFace said, TensorRT may be the most suitable approach. You can also take a look at Tensorflow Lite. They're made for different applications, but are designed to handle inference with high performance. This may not necessarily fix your issue, but it would be a good place to start.

Another, more involved approach would be to restructure your server setup. You say you need a GPU on your cloud instance, but I don't think AWS Lambda offeres GPU instances. Of course, inference is a pretty quick process so you can probably get away with a CPU under the right circumstances. You should run some tests to figure out what kind of system you need to meet your requirements (Lambda may still be appropriate) and reconsider your server setup based on that.

If it turns out you do need a GPU instance, then you can't use Lambda (at least as-is). You'll instead have to rely on EC2 instances or ECS. If that's the case, you can reduce costs through a number of ways. You can limit your GPU usage by serving the Flask app on a CPU instance (or even Lambda). Since the server itself doesn't need a GPU, this should always be the case. The server (or Lambda) can then request inference from a GPU instance when it needs it.

The GPU instance, then, needs to be setup so that it's not always running. This can be facilitated through something like Kubernetes, which will allow you to setup scaling clusters to actively meet your resource requirements. This is nice since you can address the varying amount of requests automatically with as much tolerance as you can afford. On top of this, KOPS can manage spot instances, reducing your costs further. As of now, the estimated spot price for a p2.xlarge instance is $0.2774 per hour (versus the normal $0.90 per hour with on-demand instances).

Of course, by introducing this setup, you'll still have to deal with startup time (in this case, the time it takes to start the spot instance and set things up). You can avoid this by having at least one instance constantly running, which is still going to be expensive. But this begs the question of how your service is being used and what requirements/constraints you have. For example, if user's access your server via a web site, you can have Kubernetes launch an instance when a user first access the site (with the hope being that the instance will be ready when they begin to use it). After some pre-determined amount of no activity, the instance can shut down.

If this isn't the case (e.g., people are accessing your model directly), then you're going to have to asses the cost of whatever it is you're doing and if it's worth it. GPU instances aren't cheap, and having one running on-demand is going to be costly. Using something like Lambda or scaling instances reduces this cost, but the trade-off is (and will basically always be) the slow startup time. You can't really have both. At best, you can optimize your approach and try an ensemble of methods to getting everything you need (constant up time, GPU instance, etc.) by planning scaling based on historical volume, falling back to smaller models ran on CPUs, etc., but that involves a fair amount of effort and still comes with trade-offs.

Depending on your usage, you might want to consider buying a dedicated GPU server that you can host on premise. You can even still use Lambda as your server and just have it call your local machine for inference. Again, that's not a trivial task (and it can still get expensive), but it's really not too bad, especially considering the inference specific devices that exist such as the Coral USB Accelerator ($74.99) or the NVIDIA Jetson Nano ($98.95). You could also, of course, get your hands on a cheap GPU and rig up a Linux server dedicated to inference (I'm sure you could put one together for less than $100 if you're resourceful enough). The investment you'd be making here would certainly save you money down the road if your app sees continued use.

So, with all things considered, you're at a bit of an impasse. The specific dilemma you mentioned (Tensorflow taking too long to import) can't really be resolved without a dedicated instance. You can try to reduce that overhead by avoiding importing the entire library since you only need it for inference, and utilizing TensorRT or Tensorflow Lite might do the trick. You won't be able to use Lambda for GPU inference, so you'll need to figure out if you can do without it or work with something else. If you do need GPU power, it's going to cost you no matter what. You can minimize those costs by reducing up time, which is something you need to assess. If you need 100% up time and a GPU instance, I'd consider buying a dedicated machine and avoid cloud services.

Edit: Just came across this: https://aws.amazon.com/blogs/aws/amazon-elastic-inference-gpu-powered-deep-learning-inference-acceleration/

Still doesn't solve your issue directly, but worth considering along with everything else I mentioned.

I really enjoyed Deep Learning with Python by Francois Chollet. He's the author of the Keras API in TensorFlow and the writing is very well done and very approachable. It's fairly concept heavy, but light on the math, if you then want the math a good starting place is the lecture series by Geoffrey Hinton himself, which is now available for free on YouTube. However, that series is not necessarily specific to TensorFlow. Links to both below.

​

Deep Learning w/ Python (Francois Chollet): https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438/ref=sr_1_3?crid=1VNK53QGWW12&keywords=deep+learning+with+python&qid=1562878572&s=gateway&sprefix=Deep+Learn%2Caps%2C196&sr=8-3

​

Neural Networks for Deep Learning (Geoffrey Hinton): https://www.youtube.com/playlist?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9

I'm really enjoying this book atm.

https://www.amazon.fr/Hundred-Page-Machine-Learning-Book/dp/199957950X/

I'm a big fan of the 'Deep Learning With Python' book by Francois Chollet.

https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438

​

Looks like the whole book is available here but the link is .cn so check it out on your own.

http://faculty.neu.edu.cn/yury/AAI/Textbook/Deep%20Learning%20with%20Python.pdf