• 0 Posts
  • 16 Comments
Joined 1 year ago
cake
Cake day: August 8th, 2023

help-circle




  • There’s plenty of open source projects that distribute executables (i.e. all that use compiled languages). The projects just provide checksums, ensure their builds are reproducible, or provide some other method to verify.

    In practice, you’re going to wind up in dependency hell before pypi stops hosting the package. E.g. you need to use package A and package B, but package A depends on v1 of package C, and package B depends on v2 of package C.

    And you don’t need to use pypi or pip at all. You could just download the code and directly from tbe repo, import it into your project (possibly needing to build if it has binary components). However, if it was on pypi before, then the source repo likely had all the code pip needs to install it (i.e. contains setup.py and any related files).







  • IDK, looks like 48GB cloud pricing would be 0.35/hr => $255/month. Used 3090s go for $700. Two 3090s would give you 48GB of VRAM, and cost $1400 (I’m assuming you can do “model-parallel” will Llama; never tried running an LLM, but it should be possible and work well). So, the break-even point would be <6 months. Hmm, but if Severless works well, that could be pretty cheap. Would probably take a few minutes to process and load a ~48GB model every cold start though?





  • It’s good at refactoring smaller bits of code. The longer the input, the more likely it is to make errors (and you should prefer to start a new chat than continue a long chat for the same reason). It’s also pretty good at translating code to other languages (e.g. MySQL->PG, Python->C#), reading OpenAPI json definitions and creating model classes to match, and stuff like that.

    Basically, it’s pretty good when it doesn’t have to generate stuff that requires creating complex logic. If you ask it about tasks, languages, and libraries that it has likely trained a lot on (i.e. the most popular stuff in FOSS software and example repos), it doesn’t hallucinate libraries too much. And, GPT4 is a lot better than GPT3.5 at coding tasks. GPT3.5 is pretty bad. GPT4 is a bit better to Copilot as well.


  • I haven’t ran into a good use-case to try out server-less yet. Either cold starts would be a problem (for example, I have an endpoint that needs to load a 5GB model into RAM, and it takes about 45 seconds). Or, it’s just much more expensive than a VPS if the service is projected to constantly serve many requests all day. Containerized services on a VPS doesn’t require much server maintenance (unless you have a dozen or so micro-services, then yeah, Kubernetes maintenance adds a lot of overhead).