• hendriks73 over 8 years ago

    Hey..

    for those of you who are interested in accessing Discogs images, but need more than 1,000 per day.
    I've written a simple proxy that takes care of the authorization issues. To use it, install the WAR in a servlet engine (like Tomcat) and modify your image requests (not the other API requests) to go to the proxy rather than api.discogs.com.

    The thing is licensed under GPL and comes with no warranty whatsoever.
    Code and instructions are at https://github.com/hendriks73/coxy
    A current binary as at http://www.beatunes.com/repo/maven2/com/tagtraum/coxy/0.9.0/ - use the WAR, but you might want to remove the version number from the file name, for things to be easy.

    Also, please thoroughly read the README from the github page.

    Cheers,

    -hendrik

    PS: If you are interested in "trading" the cached image folder, please contact me privately. I'm sure you can figure out a way to do so ;-)
  • Staff 22

    eknudtson over 8 years ago

    This is awesome! Thanks for making this and sharing. Proxying / caching images is a great way to interact with the API to get around the limits put in place.
  • slavoingilizov over 8 years ago

    Yes, good work. As for "trading" the cached image folder, I was wondering if the images are provided as part of the monthly data dumps [url=http://www.discogs.com/data/]. If so I guess it wouldn't be difficult to write a tool which would make them usable by Coxy.
  • hendriks73 over 8 years ago

    AFAIK, the dump does not contain the images.
    Also: 10k images are roughly equal to 1GB.
    Not sure how many images Discogs has, but let's assume millions.
    That means hundreds of GB...
  • hendriks73 over 8 years ago

    Just FYI: There is a new Coxy build at https://www.beatunes.com/repo/maven2/com/tagtraum/coxy/0.9.1/ that knows how to handle "images" instead of "image"...
  • hendriks73 over 7 years ago

    Due to recent changes in the image API (here), Coxy is currently broken.
    I'll post a fixed version soon.
  • hendriks73 over 7 years ago

    I think I fixed the issue.. required some reworking of the code, as the authentication is not necessary anymore.

    https://github.com/hendriks73/coxy/ / https://www.beatunes.com/repo/maven2/com/tagtraum/coxy/0.9.2/
  • rodneyfool over 7 years ago

    Nice work, hendriks73!
  • hendriks73 over 7 years ago

    Thanks!
    Please let me know, if you find any issues: https://github.com/hendriks73/coxy/issues
  • fheidenreich over 7 years ago

    Thanks hendriks73!

    I've just read this post from jcrosen and was wondering whether caching is still necessary to cope with the rate-limiting which now seems to be based on IP and not based on API consumer key anymore (as described in the API).

    What do you think? Is there some recommendation from the staff?
  • hendriks73 over 7 years ago

    I guess it depends on your use case... If you have not just a few, but a lot of requests by users of your app/website, you might still want to cache them in order to keep the load on discogs low.

    But yes - a staff recommendation would be helpful.
  • Staff 22

    eknudtson over 7 years ago

    eknudtson edited over 7 years ago
    We would definitely appreciate caching images as it helps to distribute the load off of our servers. It also makes sense that if you have an app generating a ton of traffic you should be helping to shoulder some (or all) of that traffic through your own infrastructure. That way you have some control over how images you use are stored/served and if we make a change on our end it won't totally throw you off.
  • HorseT over 7 years ago

    eknudtson
    We would definitely appreciate caching images as it helps to distribute the load off of our servers. It also makes sense that if you have an app generating a ton of traffic you should be helping to shoulder some (or all) of that traffic through your own infrastructure. That way you have some control over how images you use are stored/served and if we make a change on our end it won't totally throw you off.

    I totally agree with your opinion. For my project, I also created a proxy to handle the request limit Discogs provides.

    BUT: If you / Discogs appreciate it, we as developers would appreciate if you could give us some information about necessary API changes BEFORE you make them. I have to deal now with angry consumers and bad rescissions because of the changes you made.
    So please let us know about changes you planning to do in the future..
  • hendriks73 over 7 years ago

    Indeed, a little warning would be nice...
  • little_alien over 7 years ago

    HorseT
    information about necessary API changes BEFORE you make them

    Neh, Discogs has been making all kinds of changes without prior notice since 2001, so why change a tradition? ;)
  • jcrosen over 7 years ago

    HorseT
    if you could give us some information about necessary API changes BEFORE you make them.

    Yep, it's pretty clear from the feedback received in the forum and through other channels that we missed the mark we expect of ourselves. We were under some time-pressure related to failing hardware and didn't anticipate the potential problems this would cause; that's certainly on us. In particular there are two things I think we should have announced ahead of time:

    - The switch requiring authentication for database resources to get image URLs and removal of authentication required for image data
    - That the URLs would be signed and thus any modification of said URL will result in a 404 (we didn't anticipate that some OAuth libraries would cause problems like this or that apps might already be modifying the URL in some way)

    We appreciate the feedback and will incorporate it into our continuing efforts to improve the API service. We've got some additional changes upcoming and we'll do a better job of notifying y'all before the changes go live. Thanks for the discussion and your patience with this particular disruption!
  • HorseT over 7 years ago

    Sounds great ;)
  • ijabz over 7 years ago

    How does COXy work with the recent changes to images ?
  • hendriks73 over 7 years ago

    With the current Coxy version:

    Well.
  • hendriks73 over 7 years ago

    I just updated the code to be able to deal with HTTPS endpoints, as required by this recent announcement.

    You can get Coxy from GitHub.
  • hendriks73 over 7 years ago

    Note: If you are using Coxy, you might want to adjust the target.base parameter, to use https instead of http.
  • D0ctor over 6 years ago

    Hello, you can use proxy from this site: http://advanced.name. Its stability, fast and anonymouse. I use it within 2 years with great pleasure.

Log In You must be logged in to post.