The Week in AI: Plugins for ChatGPT, Everyone Else Trying To Keep Up

The pace of acceleration ramps up

There’s no way to keep up with everything happening. As soon as you read about or try one tool, ten more pop up. The through line seems to be these products are making everyday tasks infinitely easier.

So, let’s prioritize and look at a few updates worth your time and attention.

Once again, OpenAI dominated the discourse as they released plugins for ChatGPT (both native and third-party).

For the first time, these plugins give ChatGPT access to the internet and the ability to perform various tasks for you.

The first batch of plugins includes those from OpenAI, a web browser and code interpreter. 

The web browser plugin lets ChatGPT retrieve information directly from the internet. And the code interpreter plugin uses Python and performs logical calculations based on problems posed.

They’ve also open-sourced the code for a knowledge base retrieval plugin, which can be “self-hosted by any developer with information with which they’d like to augment ChatGPT.”

This technological leap has instantly made thousands of websites, jobs, and freelancers obsolete, though it’ll take the next few weeks and months for everyone to realize.

The rate of change was already at breakneck speed. And it’s accelerating. There’s not enough time and attention to grapple with it all and understand the profound changes this has already introduced.

The best way I can describe it is like this: Imagine you’re shredding on your guitar, hitting as many notes as you can with speed and virtuosity. The point is to hear and distinguish the notes. But then you start shredding so fast it all turns into one note, anyway.

It looks, feels, and sounds like history is collapsing into a never-ending now, into one long, sustained shriek.

As the plugin ecosystem develops, these apps will replace a lot of what developers, marketers, and others are doing. 

But first, let’s recap what’s going on with ChatGPT plugins

You can think of it as an iOS or Google app store, where you’ll have plugins for all sorts of tasks available to you with a click. It remains to be seen how you can monetize this, if at all (and if OpenAI will take a cut, like Apple’s 30% or Google Play’s 15-30%, or only charge for API calls).

Some of the first third-party plugins have been created:

• Shopify: Search for products.

• Instacart: Order groceries.

• OpenTable: Restaurant recommendations and booking.

• Expedia: Plan trips and book flights and accommodations.

• Wolfram: Access computation, math, and real-time data.

• Zapier: Interact and connect with the 5,000+ apps available.

In total, there are 11 of them for now:

This is another opportunity for developers to create million-dollar startups. In fact, the way you create plugins is making it simple (and easy) for anyone to create plugins at will.

All you do is use human language descriptions for everything: 

Mitchell Hashimoto @mitchellh

For those who aren't aware: you write an OpenAPI manifest for your API, use human language descriptions for everything, and that's it. You let the model figure out how to auth, chain calls, process data in between, format it for viewing, etc. There's absolutely zero glue code.

The primacy of the word is coming into view, when all you have to do is give verbal and written instructions. 

It’ll be interesting to see how this affects language and communication in ways we can’t even imagine right now.

For more details: 

• Read the full release here.

• Read the API documentation here.

• Join the waitlist here.

What’s everyone else up to these days?

There’s a lot more going on but it’s getting lost in the mix if you’re not trolling Twitter every hour of the day. Here are a handful of things you should know about:

The AI developer war continues with GitHub Copilot X, the upgraded version of Copilot:

“We are not only adopting OpenAI’s new GPT-4 model, but are introducing chat and voice for Copilot, and bringing Copilot to pull requests, the command line, and docs to answer questions on your projects.”

It’s a ChatGPT-like experience in code editors and at a glance:

• Copilot X now recognizes what code a developer has typed and the error messages shown, and it’s deeply embedded into the IDE.

• Copilot for pull requests, tags are automatically filled out by Copilot based on the changed code, and developers can review or modify the suggested descriptions.

• AI-generated answers for documentation, including questions about languages, frameworks, and technologies. 

• Copilot for the command line interface, with Copilot CLI, which can compose commands and loops, and handle obscure find flags to satisfy queries.

Google starts opening up Bard, which promises to “combine the breadth of the world’s knowledge with the power, intelligence, and creativity of our large language models.” My initial impression is that it’s as capable as ChatGPT. (Read more)

Microsoft releases an advanced Dall-E in Bing, allowing you to “create an image simply by using your own words to describe the picture you want to see. Now you can generate both written and visual content in one place, from within chat.” (Read more)

NVIDIA announces Foundations, “a set of cloud services that enable businesses to build, refine and operate custom large language models and generative AI models that are trained with their own proprietary data and created for their unique domain-specific tasks.” (Read more)

Adobe announces Firefly, “a new family of creative generative AI models, first focused on the generation of images and text effects”. Interestingly enough, Adobe will introduce a “Do Not Train” tag for creators who do not want their content used in model training. (Read more)

Canva's new AI, on top of their existing Text-to-image and Magic write features. You can now get personalized templates based on uploaded images, draw your ideas with shape recognition, align video footage to soundtrack beats, generate presentations, create animations, and much more. (Read more)

Microsoft announces Loop, which is like Notion and can “bring together teams, content, and tasks across your tools and devices.” (Read more) 

Want more? A few quick hits: 

• Bing Image Creator via Chat.

• Instruct-NeRF2NeRF

• Ubisoft's new AI tool

• Runway's new Text-to-video AI

These updates alone are big news. But announcements from big tech are drowning out each other and most niche and startup players. And with thousands of AI tools already available, finding authentic gems that offer anything different is getting harder.

Bigger players are centralizing the power of AI

Incumbents are embracing and integrating AI functions into existing form factors faster than startups can build them. 

In other words, progress is being entrenched and centralized. Startups, and ordinary users, will need to explore new ways to combine form and function if we want to exist outside of the hegemon.

We’ll see how the next few weeks and months play out. It’s going to be a wild ride.