Integrating twitch channel points and OBS

Context

As noted elsewhere on this website, I stream on twitch for fun regularly. One of the available channel point rewards is a button called "Gib meme". Once someone redeems this reward, I stop what I'm doing, open a folder, show them a funny meme, and then continue streaming. I wanted to automate this, and so here we are.

Twitch

The first thing we need to do is figure out how to know when a reward occurs. Thankfully, twitch4j exists and has pretty good documentation. It's a little confusing since twitch has 7 different APIs exposed to us to use, but looking through the site I eventually found the right one.

The documentation notes that you'll need to set up the twitch client to listen for channel point redemptions like so:

twitchClient.getPubSub().listenForChannelPointsRedemptionEvents(credential, "1234567890");
twitchClient.getEventManager().onEvent(ChannelPointsRedemption.class, System.out::println);
twitchClient.getEventManager().onEvent(RewardRedeemedEvent.class, System.out::println);

This however, assumings you have an instance of the twitch client as well as the channel id and a credential. The channel id being a string for that second argument, you might be confused and think you can just pass in your channel name, but you'd be wrong. We need the channel id, which we can through twitch's helix api, like so:

String channelId =  twitchClient.getHelix()
    .getUsers(null, null, Arrays.asList(fromChannel))
    .execute()
    .getUsers()
    .get(0)
    .getId();

This fromChannel here is your channel name, which is also just your twitch username. We still need a credential though, and this is actually where I ran into my first major roadblock. You see, twitch supports a mulitude of ways to authenticate, but, in order to use the channel point listener, I needed a client credential grant flow oauth token¹. Which, twitch4j say they support:

In particular, you can pass an app access token or user access token to withDefaultAuthToken, or specify both withClientId/withClientSecret for the library to automatically generate an app access token.

when I tried it though, I got an error. So, I ended up rolling my own credential fetcher which wasnt' too bad, and honestly gave me an excuse to use the java.net.http.HttpResponse class and its body handlers. I was also pleasantly surprised to find that javax.json provides a pretty simple-to-use JSON parser so I didn't have to go download the internet to do some basic parsing. After creating the body handler, the code became surprisingly tidy:

public class TwitchCredentialClient {
    private final HttpClient httpClient;
    transient private final String clientId;
    transient private final String clientSecret;

    public TwitchCredentialClient(String clientId, String clientSecret) {
        this.clientId = clientId;
        this.clientSecret = clientSecret;
        this.httpClient = HttpClient.newHttpClient();
    }

    public ClientCredentialFlowResponse getCredential() throws URISyntaxException, IOException, InterruptedException {
        URI providerUrl = new URI("https://id.twitch.tv/oauth2/token");

        String form = (new StringBuilder()).append("client_id=").append(clientId)
                .append("&client_secret=").append(clientSecret)
                .append("&grant_type=client_credentials").toString();

        HttpRequest httpRequest = HttpRequest.newBuilder()
                .uri(providerUrl)
                .header("Content-Type", "application/x-www-form-urlencoded")
                .POST(HttpRequest.BodyPublishers.ofString(form))
                .build();

        HttpResponse response = httpClient.send(httpRequest, new CredentialBodyHandler());
        return response.body();
    }
}

Intelliji, and maybe you, will get mad at me for not just writing "" + "" and using a string builder but I like the mental separation of here's the key and then here's the value in a somewhat explicit way. Anyway, my mini client code is used by a TimerTask class so that we can easily refresh the token whenever its expiration occurs:

public class OauthTokenTask extends TimerTask {
    Logger logger = LoggerFactory.getLogger(getClass().getCanonicalName());

    private final TokenListener tokenListener;
    private final TwitchCredentialClient twitchClient;

    public OauthTokenTask(TwitchCredentialClient twitchClient, TokenListener tokenListener) {
        this.tokenListener = tokenListener;
        this.twitchClient = twitchClient;
    }

    @Override
    public void run() {
        try {
            ClientCredentialFlowResponse credential = twitchClient.getCredential();
            logger.info("Token from twitch retrieved!");
            tokenListener.onNewCredential(credential);
        } catch (URISyntaxException | IOException | InterruptedException e) {
            throw new RuntimeException(e);
        }
    }
}

As you can see, whenever this task is ran by the scheduler, we fetch a credential and then let anyone who might need to know know. Super simple observer pattern being used here so that whenever the twitch client code requests a credential from us, we've got one that will work. What does this listener look like? I'm glad you asked. We actually have two listeners, the first is just the simple container for the information:

public class RefreshingOAuth2Credential extends OAuth2Credential implements TokenListener {
    public RefreshingOAuth2Credential(String identityProvider, String accessToken) {
        super(identityProvider, accessToken);
    }

    @Override
    public String getAccessToken() {
        return super.getAccessToken();
    }

    @Override
    public void onNewCredential(ClientCredentialFlowResponse credential) {
        setAccessToken(credential.token());
        setExpiresIn(credential.expiresInSeconds());
    }
}

Which extends the OAuth2Credential class from the twitch4j client so that we can pass it into the library. The coordinating factor for this that actually sets up the timers and wires in the above listener is itself a listener! Pay attention to the start method below, when we setup the task we pass ourselves in as the listener:

public class RefreshingOAuth2CredentialProvider implements TokenListener {
    private Timer timer;
    Logger logger = LoggerFactory.getLogger(getClass().getCanonicalName());

    private final TwitchCredentialClient twitchClient;
    OauthTokenTask lastCompletedTask;
    private RefreshingOAuth2Credential token;

    public RefreshingOAuth2CredentialProvider(TwitchCredentialClient twitchClient) {
        this.timer = getTimer();
        this.twitchClient = twitchClient;
        lastCompletedTask = null;
        this.token = null;
    }

    public void start() {
        logger.info("Starting OAuth Fetching Task");
        lastCompletedTask = new OauthTokenTask(twitchClient, this);
        timer.schedule(lastCompletedTask, 0);
    }

    public RefreshingOAuth2Credential getCredential() {
        synchronized (this) {
            return token;
        }
    }

    @Override
    public void onNewCredential(ClientCredentialFlowResponse credential) {
        if (this.token == null) {
            this.token = new RefreshingOAuth2Credential("twitch", credential.token());
        }
        // Update the token that we've given out via getCredential so it continues to work
        this.token.onNewCredential(credential);
        lastCompletedTask.cancel();
        lastCompletedTask = new OauthTokenTask(twitchClient, this);
        timer.schedule(lastCompletedTask, credential.expiresInSeconds() * 1000L);
    }
}

When we first start, we immediately fire off a task to get a credential. A moment later, once our listener method is called we'll set the token to the one we just got, and then we'll use the expiration date from said credential to decide when the task should run again. You might be curious why we're tracking the lastCompletedTask, and that's because I wanted to be able to write tests without having to Thread.sleep. And so I did, it required me to write up a fake client and mock out twitch, but the separation between my client, the observed events, and making the timer settable let me do this without having to cry everytime I ran the unit tests.

Now that I had a way to get the oauth credential, I was able to easily get my code listening to the channel events for whenever someone redeems an award:

protected TwitchClient getTwitchClient() throws IOException {
    TwitchClientBuilder builder = TwitchClientBuilder.builder()
        .withClientId(clientId)
        .withClientSecret(clientSecret)
        .withEnableHelix(true)
        .withEnableChat(true)
        .withEnablePubSub(true);
    return builder.build();
}

public void beginListeningForRewards(String fromChannel) {
    String channelId =  twitchClient.getHelix() // made during constructor via getTwitchClient()
            .getUsers(null, null, Arrays.asList(fromChannel))
            .execute()
            .getUsers()
            .get(0)
            .getId();
    twitchClient.getPubSub().listenForChannelPointsRedemptionEvents(refreshingOAuth2CredentialProvider.getCredential(), channelId);
    twitchClient.getEventManager().onEvent(ChannelPointsRedemption.class, this::redemptionHandler);
    twitchClient.getEventManager().onEvent(RewardRedeemedEvent.class, this::rewardHandler);
}

The constructor of my twitch event publisher sets up the oauth credential and an instance of my twitch credential client I noted above. When we're ready to connect, then we spinlock² until we've got the access token and then subscribe to the events via the twitch4j client:

public void connectToTwitch() throws IOException {
    refreshingOAuth2CredentialProvider.start();
    RefreshingOAuth2Credential credentials = null;
    /* Get the token from twitch on the other thread, wait here until we do */
    while (credentials == null) {
        credentials = refreshingOAuth2CredentialProvider.getCredential();
    }
    this.twitchClient = getTwitchClient();
}

Pretty easy. So, I now have a method to connect to twitch, what do we do with events? Well, the event object we're getting has a getRedemption method we can use to get the actual redemption, which has all the information you might need. For example, if you need to get the user that requested it, it's in there if the reward has input text you can get it via getUserInput(), but the only thing I really care about is the title that's nested inside of getReward's call that gets us a ChannelPointsReward.

Since I only care about the the name of the reward, I can wrap twitch up into a black box and then publish out the events via a simple String that the rest of my code can care about:

private void redemptionHandler(ChannelPointsRedemption channelPointsRedemption) {
    String id = channelPointsRedemption.getReward().getId();
    String title = channelPointsRedemption.getReward().getTitle();
    if (alreadySeenEvents.contains(id)) {
        logger.info("Event already seen: %s, skipping".formatted(channelPointsRedemption));
        return;
    }
    logger.info("Channel redemption: %s".formatted(title));
    this.newRewardEventReceived(title);
}

It's worth noting that twitch may send duplicate events but they happily provide an easy way to dedupe them via the event id. This application runs on my computer so I can restart it if I somehow run out of memory doing this, and we've got a simple HashSet to track which events we've already handled. The reward event we're publishing is just a string like I said, so the rest of my code can implement this interface:

public interface EventListener {
    void onChatMessage(ChatMessage chatMessage);
    void onReward(String reward);
}

You can ignore the chat message method for the purpose of this blog post, that's its own page I haven't written yet, but I just pass the title along and this makes it very easy to test my code without having to actually wait on setting up a connection to twitch. Now that you understand how I can go from twitch connection to twitch reward redemption, and from there to basic type for our system we can move onto the next big part of integrating twitch and obs together.

Scripting showing and hiding sources in OBS

OBS is one of the best tools to use for recording your screen for videos and also one of the most widely used open source software tools used for streaming on twitch. It's pretty extensible and also supports a web socket protocol that can be used to script basic things you could do with the software via code. This isn't how most people use OBS. Typically, you'll spend some time setting things up, maybe use streamlabs to do a couple fancy sound alert boxes or twitch chat integrations, and be content. For me, I want to give my program the power to display an image from my hard drive to the user for a period of time then hide it.

Thankfully, as is the way of things with a well established language and wide userbase, there's a Java library for working with OBS in this way. I found myself reading the sample code and README for examples but one of the most useful snippets I found for my purposes was this github issue which helped lead me to the SetInputSettings class. But I'm getting ahead of myself. Let's talk about connecting to OBS first!

The examples on the README for the java project are pretty self contained examples. And my code ended up being a simple method with the ip and password passed in. But this code right here won't actually work:

private static OBSRemoteController getObsRemoteController(String ipAddress, String password) {
    return OBSRemoteController.builder()
        .autoConnect(false)
        .host(ipAddress)
        .port(4455)
        .password(password)
        .lifecycle()
        .onReady(() -> System.out.println("OBS is ready!"))
        .and()
        .connectionTimeout(4).build();
}

The above code will only work if you enable the websocket plugin which is disabled by default:

Once I had that enabled, I was off to the races. Now, we just need a class that implements EventListener and uses this remote controller class to do stuff. But what kind of stuff? In general, anything I want to do in OBS is going to adhere to this interface:

public interface OBSTask {
    void run(OBSRemoteController obsRemoteController);
    boolean isBusy();
}

This could just as easily be an abstract class that takes the controller as a constructor argument, but I wanted to keep the remote controller a bit separated. No particular reason, as it is in most programs, I wrote it that way because I felt like it at the time. But anyway, I looked through a number of the example objects, and then through a bunch of the codebase according to my browser history.

Then I used intelliji's intellisense to explore my newly created remote controller to try to figure out how to do what I wanted. Eventually, I landed on this code to find something in my OBS scene, display it for some period of time, and then hide it again:

private void toggleSceneItemOnForDuration(OBSRemoteController obsRemoteController) throws InterruptedException {
    GetSceneListResponse sceneListResponse = obsRemoteController.getSceneList(1000);
    String sceneName = sceneListResponse.getCurrentProgramSceneName();
    GetSceneItemIdResponse sceneItemIdResponse = obsRemoteController.getSceneItemId(sceneName, getSceneItemName(), 0, 1000);
            
    // If there's no item by that name, then just stop
    if (!sceneItemIdResponse.getMessageData().getRequestStatus().getResult()) {
        getLogger().warn("No item found in scene %s by name %s".formatted(sceneName, getSceneItemName()));
        return;
    }
            
    Number itemId = sceneItemIdResponse.getSceneItemId();
    GetSceneItemEnabledResponse s = obsRemoteController.getSceneItemEnabled(sceneName, itemId, 1000);
    Boolean isEnabled = s.getSceneItemEnabled();
    if (!isEnabled) {
        obsRemoteController.setSceneItemEnabled(sceneName, itemId, true, 1000);
    } else {
        obsRemoteController.setSceneItemEnabled(sceneName, itemId, false, 1000);
        obsRemoteController.setSceneItemEnabled(sceneName, itemId, true, 1000);
    }
    // Is there a better way to do this. Sure. Do we need to?
    // No. No this is just running on my local machine. It's fine.
    Thread.sleep(getMilliBeforeDisablingItem());
    obsRemoteController.setSceneItemEnabled(sceneName, itemId, false, 1000);
}

There's a couple things that I think are worth noting from this code, put aside your pitchfork over my sleeping thread and some magic numbers tossed around. Here's what will actually help you if you're trying to do something similar:

The getCurrentProgramSceneName is how to get the currently displayed scene in OBS
Scene items are manipulated via their ID, often getting a single property has a specific helper method. So to get whether or not something is enabled or not, you need use a GetSceneItemRequest via the getSceneItemEnabled method on the controller. I'm using the synchronous versions so that 1000 is just waiting 1s for OBS to timeout, which, given that I'm running on the same system as this code, isn't going to time out in any normal operation.
If the item doesn't exist in the current scene, then the status of the item id response will be false and we should stop now. Or else we'll risk nulls.

So, armed with this code to temporarily show an already existing item in an OBS scene, it can be wrapped up into a generic task like so:

public abstract class AbstractShowThenHideTask implements OBSTask {
    abstract protected Logger getLogger();
    protected boolean busy = false;
    abstract protected String getSceneItemName();
    abstract protected long getMilliBeforeDisablingItem();

    @Override
    public void run(OBSRemoteController obsRemoteController) {
        if (busy) {
            return;
        }
        try {
            busy = true;
            toggleSceneItemOnForDuration(obsRemoteController);
        } catch (Exception e) {
            // Ensure we
            getLogger().error(e.getLocalizedMessage(), e);
        }
        busy = false;
    }

    private void toggleSceneItemOnForDuration(OBSRemoteController obsRemoteController) throws InterruptedException {
        ...
    }

    @Override
    public boolean isBusy() {
        return busy;
    }
}

And then if I already have an item in a scene I want to show, I just extend like so:

public class GiveUpRewardTask extends AbstractShowThenHideTask {
    public static String twitchRewardName = "I had bad day";

    Logger logger = LoggerFactory.getLogger(getClass().getCanonicalName());

    @Override
    protected Logger getLogger() {
        return logger;
    }

    @Override
    protected String getSceneItemName() {
        return "give up wisdom.mp4";
    }

    @Override
    protected long getMilliBeforeDisablingItem() {
        return 33 * 1000 + 500;
    }
}

The two methods at the bottom are the ones being called in the toggle item method from before. The reward name is the title of the channelpoint reward on twitch. I track this as a static property because then, in my event listener, it becomes very easy to wire the handler in like so:

private void setupTasks() {
    rewardTasks.put(QuesoRewardTask.twitchRewardName, new QuesoRewardTask());
    rewardTasks.put(GiveUpRewardTask.twitchRewardName, new GiveUpRewardTask());
    rewardTasks.put("Gib meme", new GibMemeTask());
    rewardTasks.put("Rei > Asuka", new BestGirlVoteReward("rei", bestGirlVotesDb));
    rewardTasks.put("Asuka > Rei", new BestGirlVoteReward("asuka", bestGirlVotesDb));
}

Though, as you can see I haven't done that for everything. Wups.

Even without seeing the rest of that class you can probably tell what's going on here. We've got a map from twitch reward to OBSTask. If you expect that run method of that interface to be called by the EventListener's onReward implementation, you'd be right!

@Override
public void onReward(String reward) {
    if (hasTaskFor(reward)) {
        OBSTask task = rewardTasks.get(reward);
        if (!task.isBusy()) {
            executorService.submit(() -> task.run(obsRemoteController));
        }
    }
}

But anyway, let's get back to the OBS related code and manipulating items, because there's one thing that's worth talking about here that isn't a straightforward thing. We can enable and disable an item in a scene, but how do we adjust properties? As I stated at the top, I want to automate giving someone a random image from my hard drive when they redeem a "Gib meme" reward.

Each item in OBS can be viewed as an input of some kind (besides outputs, but let's put that outside), And each input has a set of properties. These can be manipulated using the SetInputSettings class. The most interesting thing about this though is that it wasn't really clear to me how to tell what settings to set. So, I did a quick little debugger session:

As you can see, it's a JSON object under the hood. Specifically, for the video input source, the file itself is shown as "Local File" in the UI, but in the JSON object it's local_file and a string. When I repeated this experiment for an image source (since I'm only automatically showing people images from my system), the json object included a file field for the input source. Which didn't adhere to the pattern I saw with the video input. Local File -> local_file, but Image File -> file. So, I'd suggest looking into the protocol documentation or running your debugging with a GetInputSettings request for your input to see what sort of things you should be setting. The keys are case sensitive by the way, if you set a text property when the property name is actually Text, it won't change anything in OBS.

So, with all of that in mind, it's actually really simple to change the image of an already existing source in OBS:

private void swapImageWithNewFile(OBSRemoteController obsRemoteController, Path path) {
    JsonObject fileObject = new JsonObject();
    fileObject.addProperty("file", path.toAbsolutePath().toString());
    SetInputSettingsResponse response = obsRemoteController.setInputSettings(inputName, fileObject, true, 1000);
}

That true for the 3rd parameter is really important though:

This controls whether our json object replaces the entirety of the settings for the given input, or if it's merged on top of them. Since I'm only changing the file value and I want to keep the other settings, including the resizing and other possible transformations I've done (so the images won't take up the entire screen when shown), we need overlay to be true so we patch the values in.

So, I now have the ability to show an input for a period of time, and change the input's source at will. The rest of the code to actually find the memes on the file system is plain old java with no OBS in site.

public class MemeFileSelector {
    private final Path folder;
    MemeFilter memeFilter;

    public MemeFileSelector(Path memeFolder) {
        this.folder = memeFolder;
        this.memeFilter = new MemeFilter();
        // If I want to persist memes shown to chat across streams, then
        // we'd want to load those files in here.
        this.memeFilter.markMemeSeen(getNoMemeToShowMeme());
    }

    private @NotNull Path getNoMemeToShowMeme() {
        return folder.resolve("nice-boat.jpg");
    }

    public Path getRandomUnseenFile() {
        File[] files = folder.toFile().listFiles(this.memeFilter);
        if (files.length == 0) {
            return getNoMemeToShowMeme();
        }

        int randomIndex = ThreadLocalRandom.current().nextInt(0, files.length);
        if (randomIndex < files.length) {
            File file = files[randomIndex];
            memeFilter.markMemeSeen(file.toPath());
            return file.toPath();
        } else {
            // ??? Shouldn't get here but if so...
            return getNoMemeToShowMeme();
        }
    };
}

Normally I use a FileVisitor for traversing directories, but since I'm only planning on having 1 level to this directory, the listFiles method seemed like a good choice. The only thing of note here, besides the boat default image, is that we don't want to show the same image twice in one stream because that'd be no fun if someone redeemed their points and didn't get something new. So, a simple file name filter will do the trick:

public class MemeFilter implements FilenameFilter {
    Set<Path> seenToday;

    public MemeFilter() {
        this.seenToday = new HashSet<>();
    }

    public void markMemeSeen(Path path) {
        this.seenToday.add(path);
    }

    public boolean accept(File dir, String name) {
        boolean isJpg = name.endsWith(".jpg");
        boolean isPng = name.endsWith(".png");
        boolean isGif = name.endsWith(".gif");
        boolean isImage = isGif || isJpg || isPng;

        Path resolved = dir.toPath().resolve(name);
        boolean notSeen = !seenToday.contains(resolved);

        if (isImage && notSeen) {
            return true;
        }
        return false;
    }
}

Which is really just a wrapper around a set that the previous code calls after the caller has asked us for a file. It also makes sure we only grab images as well, but the main thing is to make sure we don't have a repeat file.

@Override
public void run(OBSRemoteController obsRemoteController) {
    busy = true;
    try {
        Path meme = memeSelector.getRandomUnseenFile();
        getLogger().info("Showing meme %s".formatted(meme.toAbsolutePath()));
        swapImageWithNewFile(obsRemoteController, meme);
        // Then show it for Ns, then hide it.
        ShowMemeTask task = new ShowMemeTask();
        getLogger().info("Running task to show meme");
        task.run(obsRemoteController);
    } catch (Exception e) {
        getLogger().error(e.getLocalizedMessage(), e);
    }
    busy = false;
}

Our OBS Task uses the code I've mentioned to get a random image we haven't seen, then swaps the file in OBS, then uses our other OBS task to show the result for a period of time. Pretty straightforward!

Wrap up

This post isn't really a full tutorial on how exactly to set this up yourself. But I figured it would be useful to point out a couple little pain points I had with both the twitch API for a local application with no server, as well as how to update the image input sources in OBS programmatically. If you want to see the full code to get some ideas or for reference, it's up on github here. There's some other fun stuff in there, like a chat window in swing, and a method for people to vote for the 20+ years debate between who's better: Rei or Asuka from Evangelion. Enjoy!

Footnotes

The app acess token is meant for server side applications. So if you're reading this and making something that runs on the client side, don't use it. You don't want to expose your client secret to the world. In my case, this application runs on my local machine, so a remote server oauth like most of the authentication options twitch offers are aren't on the table. We need machine-to-machine conversation between our program and twitch.
↩
Spinlocks aren't normally a good thing to do, if I wanted to eliminate this we could separate things out a bit or directly call the task to get the credential first, but I didn't want to do something outside of the scheduler. So I opted for the simple spinlock since, like a lot of things in this project, this is just code running on my computer for my channel, so spinning my cpu for a moment while we send an http request to twitch and get a token back isn't the end of the world to me. ↩