Ginzoyel
Ginzoyel4mo ago

Bringing in 1-2 weeks of normal chat messages (banter)

Having this at our own URL. We'd like to support/display Discord channels that are not forums. Blacklisting them in robots.txt so these ones are not indexed but still show the community 'banter' from the past 1-2 weeks. Mostly as a way to show users that we have a fun and active community. We would not do this for many channels, but a few of those which have awesone discussion. We'd still use AO primarily for SEO. Is this at all possible, maybe with some changes to the bot? Please note that I would be more than happy to supply the feature myself via PR to Github. I just need to know what is already possible with the bot, as I assume now we catch only threads and not individual messages. Note that to avoid any rate-limiting or complaints on Discord's side, we'd only index a fair amount of regular channel messages/banter. Please see photo for an example of the banter/regular messages we want to display.
No description
67 Replies
Rhys
Rhys4mo ago
It is possible w/ the bot it's more a matter of displaying them, I can work something out here
Ginzoyel
GinzoyelOP4mo ago
I love you. You're going to get the fattest donation to Github 😄 our company is begging for this. Hey mate. Kust checking in, we'e fully moved over to AO now from Linen and my team keeps asking me when this will be done. This is a pretty high prio thing for us, is there something we can do to incentivise this? Can I help out in someway? I'm a full time backend developer.
Rhys
Rhys4mo ago
Getting subpaths working for you now, will follow up on the making regular text channels displayed after Update here, have subpaths mostly working I see you've got an existing workaround for it that's a bit hacky, once this is ready it'll also do canonical urls for you and fix being able to access the host domain directly no action needed from you yet, once its ready i'll probably have you swap out your existing script Ok I've got subpaths working now with a proper setup The script you'll need to put in your Cloudflare worker is:
addEventListener("fetch", (event) => {
event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
const SUBPATH = "/community";

try {
const urlObject = new URL(request.url);
// If the request is to the subdirectory
if (new RegExp(`^${SUBPATH}`).test(urlObject.pathname)) {
// Then Proxy to Answer Overflow
const DOCS_URL = "community.migaku.com";
const CUSTOM_URL = "migaku.com";

let url = new URL(request.url);
url.hostname = DOCS_URL;

// Strip SUBPATH from the pathname
url.pathname = url.pathname.replace(new RegExp(`^${SUBPATH}`), "");
// If the path becomes empty, set it to "/" to avoid issues with some servers
if (url.pathname === "") {
url.pathname = "/";
}

let proxyRequest = new Request(url, request);

proxyRequest.headers.set("Host", DOCS_URL);
proxyRequest.headers.set("X-Forwarded-Host", CUSTOM_URL);
proxyRequest.headers.set("X-Forwarded-Proto", "https");
proxyRequest.headers.set("X-AnswerOverflow-Skip-Subpath-Redirect", "True"); // Prevents being redirected to the main url
return fetch(proxyRequest);
}
} catch (error) {
// if no action found, play the regular request
const newRequest = new Request(request);
return await fetch(newRequest);
}
// If no action found and no error, play the regular request (with the added header)
const newRequest = new Request(request);
return await fetch(newRequest);
}
addEventListener("fetch", (event) => {
event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
const SUBPATH = "/community";

try {
const urlObject = new URL(request.url);
// If the request is to the subdirectory
if (new RegExp(`^${SUBPATH}`).test(urlObject.pathname)) {
// Then Proxy to Answer Overflow
const DOCS_URL = "community.migaku.com";
const CUSTOM_URL = "migaku.com";

let url = new URL(request.url);
url.hostname = DOCS_URL;

// Strip SUBPATH from the pathname
url.pathname = url.pathname.replace(new RegExp(`^${SUBPATH}`), "");
// If the path becomes empty, set it to "/" to avoid issues with some servers
if (url.pathname === "") {
url.pathname = "/";
}

let proxyRequest = new Request(url, request);

proxyRequest.headers.set("Host", DOCS_URL);
proxyRequest.headers.set("X-Forwarded-Host", CUSTOM_URL);
proxyRequest.headers.set("X-Forwarded-Proto", "https");
proxyRequest.headers.set("X-AnswerOverflow-Skip-Subpath-Redirect", "True"); // Prevents being redirected to the main url
return fetch(proxyRequest);
}
} catch (error) {
// if no action found, play the regular request
const newRequest = new Request(request);
return await fetch(newRequest);
}
// If no action found and no error, play the regular request (with the added header)
const newRequest = new Request(request);
return await fetch(newRequest);
}
If you're fine with downtime you can put that in now and when I'm up I'll make the swap over, otherwise we can do it tomorrow with 0 / less downtime
Rhys
Rhys4mo ago
Demo of it working over at https://rhys.ltd/idk While I do love the inguity of getting subpaths working with your workaround this sets up a few things more correctly, especially with the api & analytics
Search Engine Test V2
Search Engine Test V2 Community
View the Search Engine Test V2 Discord server on the web. Browse questions asked by the community and find answers.
Ginzoyel
GinzoyelOP4mo ago
Perfect! So in AO do we still use community.oursite.com and then a Cloudflare Worker or how does it work? I tried oursite.com/community in AOs' DNS but nothing happened. It seems only a subdomain is supported. We were already using a Cloudflare Worker in the meantime but the metadata for everything is messed up and we don't want to waste our crawl budget.
Rhys
Rhys4mo ago
Keep the community domain set how it is, I'll update it on my end
Ginzoyel
GinzoyelOP4mo ago
No description
No description
Rhys
Rhys4mo ago
Yeah the metadata should be fixed with this approach
Ginzoyel
GinzoyelOP4mo ago
On the left is the results from the Cloudflare Worker (no metadata, bad thread names etc) Alright I'll jump on this immediately.
Rhys
Rhys4mo ago
Let me know when you have this set on your worker and I'll update it on my end as well
Ginzoyel
GinzoyelOP4mo ago
Will do. Done! Please update it on your end 🙏 and thanks A LOT! As promised you now have the monthly sponsor 😄
Rhys
Rhys4mo ago
Updating now Cool looks to be working, :ty: for the sponsor also! Will polish it over the next few days for anything that pops up, also will get started on displaying regular channels
Ginzoyel
GinzoyelOP4mo ago
Pleasure mate. Thank you so much. We can drop you a few for the remaining features. You have no idea how much we appreciate this. @Rhys Not sure if this is premature but I don't seem to be able to login to our Answer Overflow account.
Rhys
Rhys4mo ago
Ah yeah this is on the migaku.com/community page?
Ginzoyel
GinzoyelOP4mo ago
Yep
Rhys
Rhys4mo ago
That’s expected, I need to hide the button Login in on those pages doesn’t really do anything at the moment - it will in the future but atm it’s fine to just hide no features will be lost
Ginzoyel
GinzoyelOP4mo ago
Ok no problem. Looks to be working otherwise.
Rhys
Rhys4mo ago
Will be back in a few hours and will fix it then
Ginzoyel
GinzoyelOP4mo ago
Only thing I can see now is that in place of obfuscated user images, the page renders a tiny copy of the entire root domain i.e. https://oursite.com
Ginzoyel
GinzoyelOP4mo ago
We solved this previously, here was our worker script: https://pastebin.com/mbRYMXGh
Pastebin
CF Worker - Pastebin.com
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Ginzoyel
GinzoyelOP4mo ago
No description
Ginzoyel
GinzoyelOP4mo ago
Thanks for also removing user page links for obfuscated users!
Rhys
Rhys4mo ago
will fix!
Rhys
Rhys4mo ago
@Ginzoyel did you change anything on the worker script? The https://migaku.com/community page isn't resolving to answer overflow
404
This content is not available on this domain.
Rhys
Rhys4mo ago
Trying to figure out if it's my end or yours, looking on my end now
Ginzoyel
GinzoyelOP4mo ago
We're just using the Worker Script you posted before. Nothing else. It was working a few minutes ago 😮 Nothing else changed on our end
Rhys
Rhys4mo ago
gotcha, taking a look at what might have changed on my end
Ginzoyel
GinzoyelOP4mo ago
So we've got nothing set up at the moment except the Worker Script you gave. All the other hacky stuff and the previous one I sent you on Pastebin we ditched. Just shared the Pastebin one so you could see how we handled obfuscated user images as those weren't resolving. I'll check our end in the meantime but I'm 99% sure we've touched nothing since swapping out the Worker Script.
Rhys
Rhys4mo ago
:ty: let me know if you see any errors i.e things being blocked in the workers script, could be requests are getting blocked somehow although it doesnt look like it Sorry about the rough edges first time doing subpaths, once we get these worked out it should be smooth sailing 🙇‍♂️ I think I know what's happening :loading:
Rhys
Rhys4mo ago
I think Cloudflare is blocking your requests, in your DNS settings for https://community.migaku.com/ turn off the Cloudflare proxy
404
This content is not available on this domain.
Rhys
Rhys4mo ago
No description
Ginzoyel
GinzoyelOP4mo ago
Done:
No description
Rhys
Rhys4mo ago
That shouldn’t be an A record, it should be a CNAME record set to cname.answeroverflow.com
Ginzoyel
GinzoyelOP4mo ago
Ok give me a moment.
Ginzoyel
GinzoyelOP4mo ago
No description
Ginzoyel
GinzoyelOP4mo ago
This okay?
Rhys
Rhys4mo ago
Yeah that looks right
Ginzoyel
GinzoyelOP4mo ago
Done Still getting a 404.
Rhys
Rhys4mo ago
The block that I was seeing is gone so that's good, although yeah it's still 404ing
Ginzoyel
GinzoyelOP4mo ago
I guess I don't need to touch anything on the AO dashboard?
Rhys
Rhys4mo ago
No nothing there would impact it I didn't change anything on my end also from when it was working, so I'm not quite sure where the problem is
Ginzoyel
GinzoyelOP4mo ago
DNS Propagation Checker - Global DNS Checker Tool
Instant DNS Propagation Check. Global DNS Propagation Checker - Check DNS records around the world.
Rhys
Rhys4mo ago
I don't see requests coming in when I load the /community route and loading the subdomain with the bypass header loads the content Can you add a header to the response from the Cloudflare worker?
Ginzoyel
GinzoyelOP4mo ago
We ultimately want to hide community.migaku.com entirely from Google Search, using a 301 Redirect or something else. and use migaku.com/community Sure, anything in particular?
Rhys
Rhys4mo ago
Basically to this add some indicator Lets do x-proxy-status-code and then the result code of the fetch call Yeah it's setup to do that, if you go to community.migaku.com you'll see you get redirected to migaku.com/community with a 307 redirect - I can change that to 308 once we've got everything fully setup
Ginzoyel
GinzoyelOP4mo ago
That's great. I'll get the header added in, hang on. Ok here is the new worker.
addEventListener("fetch", (event) => {
event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
const SUBPATH = "/community";

try {
const urlObject = new URL(request.url);
// If the request is to the subdirectory
if (new RegExp(`^${SUBPATH}`).test(urlObject.pathname)) {
// Then Proxy to Answer Overflow
const DOCS_URL = "community.migaku.com";
const CUSTOM_URL = "migaku.com";

let url = new URL(request.url);
url.hostname = DOCS_URL;

// Strip SUBPATH from the pathname
url.pathname = url.pathname.replace(new RegExp(`^${SUBPATH}`), "");
// If the path becomes empty, set it to "/" to avoid issues with some servers
if (url.pathname === "") {
url.pathname = "/";
}

let proxyRequest = new Request(url, request);

proxyRequest.headers.set("Host", DOCS_URL);
proxyRequest.headers.set("X-Forwarded-Host", CUSTOM_URL);
proxyRequest.headers.set("X-Forwarded-Proto", "https");
proxyRequest.headers.set("X-AnswerOverflow-Skip-Subpath-Redirect", "True"); // Prevents being redirected to the main url
const proxiedRes = await fetch(proxyRequest);
return withStatusHeader(proxiedRes);
}
} catch (error) {
// if no action found, play the regular request
const newRequest = new Request(request);
return await fetch(newRequest);
}
// If no action found and no error, play the regular request (with the added header)
const newRequest = new Request(request);
const originRes = await fetch(newRequest);
return withStatusHeader(originRes);
}

function withStatusHeader(res) {
const headers = new Headers(res.headers);
headers.set("x-proxy-status", res.status.toString());
return new Response(res.body, {
status: res.status,
statusText: res.statusText,
headers,
});
}
addEventListener("fetch", (event) => {
event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
const SUBPATH = "/community";

try {
const urlObject = new URL(request.url);
// If the request is to the subdirectory
if (new RegExp(`^${SUBPATH}`).test(urlObject.pathname)) {
// Then Proxy to Answer Overflow
const DOCS_URL = "community.migaku.com";
const CUSTOM_URL = "migaku.com";

let url = new URL(request.url);
url.hostname = DOCS_URL;

// Strip SUBPATH from the pathname
url.pathname = url.pathname.replace(new RegExp(`^${SUBPATH}`), "");
// If the path becomes empty, set it to "/" to avoid issues with some servers
if (url.pathname === "") {
url.pathname = "/";
}

let proxyRequest = new Request(url, request);

proxyRequest.headers.set("Host", DOCS_URL);
proxyRequest.headers.set("X-Forwarded-Host", CUSTOM_URL);
proxyRequest.headers.set("X-Forwarded-Proto", "https");
proxyRequest.headers.set("X-AnswerOverflow-Skip-Subpath-Redirect", "True"); // Prevents being redirected to the main url
const proxiedRes = await fetch(proxyRequest);
return withStatusHeader(proxiedRes);
}
} catch (error) {
// if no action found, play the regular request
const newRequest = new Request(request);
return await fetch(newRequest);
}
// If no action found and no error, play the regular request (with the added header)
const newRequest = new Request(request);
const originRes = await fetch(newRequest);
return withStatusHeader(originRes);
}

function withStatusHeader(res) {
const headers = new Headers(res.headers);
headers.set("x-proxy-status", res.status.toString());
return new Response(res.body, {
status: res.status,
statusText: res.statusText,
headers,
});
}
Ginzoyel
GinzoyelOP4mo ago
Do I still need to keep this
No description
Ginzoyel
GinzoyelOP4mo ago
for the new worker to do its job?
Rhys
Rhys4mo ago
Uh, not sure what that one is Is that the old one? Maybe that’s the one that’s swallowing the routes
Ginzoyel
GinzoyelOP4mo ago
I'll delete it. I must have missed it. We needed a ton of stuff for it to work before and my boss wanted it now now now 😄
Rhys
Rhys4mo ago
Sorry about the delay 😔 have been very busy with work
Ginzoyel
GinzoyelOP4mo ago
Hey seriously mate no problem you are saving our bacon here. We're a startup and dunno if you have worked for one but they always want everything 'yesterday' 😄
Rhys
Rhys4mo ago
I did a test deployment to https://rhys.ltd/community and it works on that one so my gut feeling is it’s something on your end - still investigating to see though
Migaku
Migaku Community
Migaku is the fastest way to really learn a language! https://migaku.com
Ginzoyel
GinzoyelOP4mo ago
Yeah I saw that one. I think it's our Cloudflare. I thought I deleted everything, I'll dump this one as well. Only other thing I saw with yours yeah was that the images for obfuscated names didn't work when you enabled them. Ah okay sorry that was an old tab of mine, I mean this:
Ginzoyel
GinzoyelOP4mo ago
No description
Ginzoyel
GinzoyelOP4mo ago
is gone. But our worker isn't being triggered by anything. I was using that^ to trigger the worker before. How are you triggering yours?
Rhys
Rhys4mo ago
I added the domain to the worker this might work for triggering it though as well
Rhys
Rhys4mo ago
Mintlify
Cloudflare - Mintlify
Host documentation at a /docs subpath using Cloudflare Workers
Ginzoyel
GinzoyelOP4mo ago
No description
Ginzoyel
GinzoyelOP4mo ago
Got it like this now and it's already working again. Just the issue with obfuscated username images that I can see off the top of my head.
Rhys
Rhys4mo ago
Sounds good, I can fix that one now was just holding off on changes until the big one was fixed
Ginzoyel
GinzoyelOP4mo ago
So yeah in short, we had old proxy blocking everything and then nothing to trigger the new worker.
Rhys
Rhys4mo ago
fixed
Ginzoyel
GinzoyelOP4mo ago
Champion Would be great if we could have that 308 Permanent Redirect asap, our Google guy is going to try and get everything removed which is from community.migaku.com using the removal tool.
Rhys
Rhys4mo ago
done
Ginzoyel
GinzoyelOP4mo ago
Looks good on our end! WOOT! It seems there aren't any messages rolling in to our AO server after 3.6.2025? We've had a few posts since then. None are showing up. @Rhys sorry to bother you mate, but could you do a quick resync or something? No idea why it has stopped syncing for us. We've got nothing newer than 03.06.2025 syncing anymore. I signed in again to get the bot authorised but that didn't seem to trigger anything even after some hours. Don't mean to distract you from working on the text channel/banter support. Hope it's not too difficult to force a refresh.
Rhys
Rhys4mo ago
syncing should just work, don't kick the bot / change permissions as that might make things worse / break things i've manually set it to first in the queue and started that to rerun, will check back in

Did you find this page helpful?