Leaked details of GPT-4

Plus: Analyze documents with AI

July 11, 2023

Hey folks,

In today’s update:

📰 News: Leaked details of GPT-4 architecture.

🛠️ Tools: Better way to analyze documents.

🎲 Misc: AI could predict Parkinson’s 7 years before clinical diagnosis.

Reading time is 4 minutes. Let’s dive in 🤿

 STARTUPS

GPT-4 Architecture leaked

One thing OpenAI hasn’t been open about is GPT-4. There have been wide speculations on why OpenAI is quiet on how the AI model works under the hood. For the first time, we’re getting a look at how OpenAI trained the model.

Here are the key details:

  • Size: 10x the size of GPT-3 with estimated 1.8 trillion parameters, trained on 13 trillion tokens!

  • Mixture of Experts (MoE): Instead of one large model, GPT-4 follows MoE strategy with 16 experts

  • Training cost: ~$63 million

  • Inference cost: 3x of GPT-3.5

Bigger Picture: $63 million to train an AI model might seem huge, but it pales in comparison to the billions of dollars being poured in building AI and supercomputers. Just last week, Inflection announced a $800 million supercomputer and its only the beginning. Read more about the architecture.

 LEVERAGE AI

Remove distractions from multiple apps

Spoke lets you aggregate, prioritize, and summarize across your tools to create the context for work. It helps you understand long threads and busy channels in slack, and summarizes your notifications across tools along with helping you with follow ups.(link)

Analyze documents with AI

MapDeduce makes document analysis simple! It handles, summarizes, analyzes and dives into the your complicated documents, spots potential red-flag terms in a contract and you could even ask questions about the document.(link)

 MISC

What’s happening in Tech

  • Healthcare: Parkinson's can be accurately predicted by motion data from smartwatches, AI models could see signs as much as seven years before clinical diagnosis! - IEEE

  • Environment: Wildfire detection startup Pano AI extends its $20M Series A with another $17M - Techcrunch

  • Space: SpaceX is re-using the first stage Falcon 9 rocket booster for 16th time! -  Ars Technica

  • Privacy: Your mobile location data is being sold to highest bidder, but a new law could soon put a total ban on buying and selling - WSJ

  • Robotics: NASA’s humanoid robot is heading to work as remote oil rig attendant - The Register

  • Layoffs: Tech layoffs are not over yet, ~300 employees laid off at Microsoft - CNBC