• fheidenreich over 6 years ago

    I'm using the Discogs API via OAuth for quite some time and it worked flawlessly since a few days ago where users of my application started to getting errors "401 unauthorized" when searching via the search endpoint.

    The interesting bit is, that not all searches are giving this error, but only those where two or more words are used for the search. To illustrate,
    http://api.discogs.com/database/search?release_title=nevermind
    gives a correct result but
    http://api.discogs.com/database/search?release_title=in+utero
    gives the above-mentioned error.

    I've investigated a bit and noticed that if I encode the space by %20 and do not use the + sign (which I've used for a very long time now), the multi-word search also gives a correct result. Confusingly, when I the revert back to use the + separator, it also works.

    Can please anyone from the staff shed some light onto this?

    Thanks!

    Florian
  • rodneyfool over 6 years ago

    Hi fheidenreich,

    Hmm, I am not able to duplicate this issue; can you post all request headers you are sending (including user agent) and your request method (e.g., GET). That will help me troubleshoot this issue.

    Thanks,
    Rodney
  • SchusterBach over 6 years ago

  • rodneyfool over 6 years ago

    Thanks, I'm trying to duplicate this response now.

    Just a side note to everyone experiencing this issue: be sure that your oauth_timestamp is dynamically generated at the time of the request; the OAuth server we use returns 401s if these timestamps are too old (5 minutes, I believe). Also, a note about the nonce value:

    "The Consumer SHALL then generate a Nonce value that is unique for all requests with that timestamp."

    source: http://oauth.net/core/1.0/#nonce

    As I'm sure this is not the solution to all of the 401s you all have been seeing lately, I'm still looking into other potential causes/triggers.

    Thanks,
    Rodney
  • p.sanders over 6 years ago

    p.sanders edited over 6 years ago
    Rodney, another data point for you. As of a few minutes ago, I am experiencing error 401's (reliably) when searching via https, but not when searching via http. My customers however, those that are affected, are using http.

    Here are details of a typical failing request (captured by my network sniffer over http, but the requests sent over https are the same and were previously working):

    GET /database/search?artist=Paul%20Simon&oauth_consumer_key=[redacted]&oauth_nonce=141383724281&oauth_signature=[redacted]&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1413837242&oauth_token=[redacted]&oauth_version=1.0&per_page=100&release_title=Graceland&type=release HTTP/1.0
    Accept: */*
    Host: api.discogs.com
    User-Agent: VinylStudio
  • little_alien over 6 years ago

    Strangely I am also unable to replicate the 401's. I've recently switched my Node.js client library to https after some playing around with it and everything seems to functioning normally. I haven't deployed it in a real production environment yet though...
  • fheidenreich over 6 years ago

    Hi rodneyfool

    Many thanks for your feedback and for investigating this issue.

    I collected some data, here is one request that is working fine:

    Sent Headers:
    GET /oauth/identity?oauth_consumer_key=[...]&oauth_nonce=1413873581110&oauth_signature=[...]&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1413873581&oauth_token=[...]&oauth_version=1.0 HTTP/1.1
    Host: api.discogs.com
    User-Agent: Mp3tag/2.65
    Accept:*/*
    Accept-Encoding: gzip

    Received Headers:
    HTTP/1.1 200 OK
    Reproxy-Status: yes
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate
    Content-Type: application/json
    Content-Length: 130
    Connection: close
    Date: Tue, 21 Oct 2014 06:39:41 GMT
    Server: lighttpd


    And here is one that is not working:
    Sent Headers:
    GET /database/search?oauth_consumer_key=[...]&oauth_nonce=1413873585172&oauth_signature=[...]&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1413873585&oauth_token=[...]&oauth_version=1.0&release_title=In+Utero HTTP/1.1
    Host: api.discogs.com
    User-Agent: Mp3tag/2.65
    Accept:*/*
    Accept-Encoding: gzip

    Received Headers:
    HTTP/1.1 401 Unauthorized
    Reproxy-Status: yes
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate
    Content-Type: application/json
    WWW-Authenticate: OAuth realm="http://api.discogs.com"
    Content-Length: 61
    Connection: close
    Date: Tue, 21 Oct 2014 06:39:45 GMT
    Server: lighttpd
  • zonesny over 6 years ago

    Hi Rodney,
    I'm also experiencing this issue, and ONLY while performing database searches. For me, I first started seeing the problem Thursday night (10/16/14), if that's any help. I have no difficulty with "identity" or "image" requests. I'm using the PHP Tonefolder classes (https://gist.github.com/tonefolder/44191a29454f9059c7e6). I'm testing everything locally.

    USER AGENT is as follows:
    SameThatTuneDiscogsClient/1.0 +http://samethattune.com

    Identity (successful)

    http://api.discogs.com/oauth/identity?oauth_consumer_key=[MYKEY]&oauth_nonce=ENRF7&oauth_signature=[MYSIGNATURE]&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1413911901&oauth_token=[MYTOKEN]&oauth_version=1.0

    Database Search (failure)

    http://api.discogs.com/database/search?q=The%20Police&type=artist?oauth_consumer_key=[MYKEY]&oauth_nonce=ENRF7&oauth_signature=[MYSIGNATURE]&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1413911901&oauth_token=[MYTOKEN]&oauth_version=1.0

    When the request fails, I also receive the following 401 message:

    "You must authenticate to access this resource."

    Please let me know if I can provide any other helpful information. Thanks.
  • rodneyfool over 6 years ago

    fheidenreich,

    In addition to the PM I sent you, it also looks like your search terms are being sent in the OAuth verifier field. Be sure to double check that all values are being sent with the correct keys.

    Thanks,
    Rodney
  • bedhed3000 over 6 years ago

    zonesny
    I'm also experiencing this issue, and ONLY while performing database searches. For me, I first started seeing the problem Thursday night (10/16/14), if that's any help. I have no difficulty with "identity" or "image" requests. I'm using the PHP Tonefolder classes (https://gist.github.com/tonefolder/44191a29454f9059c7e6). I'm testing everything locally.


    It seems like certain OAuth libraries are failing. The OAuthSimple library was changed a few weeks ago, and that broke any and all authorization involving discogs' API. I reverted the changes and made a fork (see the link above) to get it working again. Now, as zonesny mentions, it works for everything but the search endpoint. I am going to try a few other libraries to see if I can find a common thread among the one's that do not work.
  • p.sanders over 6 years ago

    The search endpoint is the one that I also am having trouble with. I've no reason to suspect my OAuth library (liboauthcpp) as most searches succeed and the problem seems to be specific to just a few customers' machines. Furthermore, customers have reported that the problem sometimes goes way of its own accord. It's all rather mysterious.
  • bedhed3000 over 6 years ago

    Definitely not a problem with the library itself, but rather issues on discogs' end that are making requests using certain libs get rejected. Just guessing. I am trying little_alien's nodejs client to see if I can get it working.
  • zonesny over 6 years ago

    Thanks Rodney and bedhed3000. I am passing the following keys in the database search example above:

    <code>
    $params['path'] = $query; //the query string containing encoded key/value pairs (e.g. 'q', 'type', 'per_page')
    $params['signatures'] = $oauth_props; //$oauth_props contains 'oauth_token' and 'oauth_token_secret' from access request
    </code>

    Not sure how the search terms would have been placed in the OAuth verifier field? What does the Discogs API expect for the 'signatures' values? Maybe in OAuthSimple's _normalizedParameters() function the field is being merged?
  • rodneyfool over 6 years ago

    zonesny
    Not sure how the search terms would have been placed in the OAuth verifier field?

    Oh no, this issue seemed to be specific to another user whose requests were showing up in the error logs.

    zonesny
    What does the Discogs API expect for the 'signatures' values?

    FWIW, we use the popular Python oauth2 library, and it expects either plaintext or HMAC-SHA1 for a signature. You can read about these signing methods here: http://oauth.net/core/1.0/#signing_process

    As always, we recommend users use an OAuth client library to abstract most of the OAuth stuff away (you'll usually just have to plugin a few OAuth endpoints and your app key and secret).
  • zonesny over 6 years ago

    Thanks, my bad Rodney. I now see the Verifier comment was directed to fheidenreich. I agree it would be best to use an existing client. However, I can't use PHP >=5.4.0, as required by ricbra's php client (I'd like to use either a Java or PHP client/library).

    Does the Discogs Python Client work correctly for ALL requests (i.e. identity, image, database search)? If so, perhaps we should compare headers with the other clients/libs that are failing? To bedhed's point, it seems like several libraries are failing with the database search (p.sanders mentioned using a C++ library, I'm using PHP), which would indicate an issue with the API.
  • CarlH over 6 years ago

    I take it this is the correct place to continue the topic of OAuth and 401 errors. I am using gtm-oauth for OAuth in a client application that runs on iOS and I have not been able to get past the 401 errors. Unfortunately, I have no clue as to where the problem is since there I have no visibilty into what the problem is other than a generic 401 error. Any assistance in troubleshooting this would be greatly appreciated. Just let me know what information is needed.

    In case it is any help here is an example identity request:
    GET /oauth/identity HTTP/1.1
    Host: api.discogs.com
    Content-Type: application/x-www-form-urlencoded
    Cookie: __utma=15419939.1949955997.1413851684.1413851684.1413851684.1; __utmz=15419939.1413851684.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _ga=GA1.2.1949955997.1413851684; ck_username=CarlH; session="[redacted]_expires=STE0MTUwNjQwNTcKLg==&auth_token=[redacted]&created_at=Y2RhdGV0aW1lCmRhdGV0aW1lCnAwCihTJ1x4MDdceGRlXG5ceDE1XHgwMVx4MTQ5XHgwNVx4ZmNceDE3JwpwMQp0cDIKUnAzCi4="; sid=1e41bfe9777725dec62bb7e800aae1a6
    Connection: keep-alive
    Accept: */*
    User-Agent: MyVinyl/2.3.2 +http://www.carlhaines.com/myvinyl
    Authorization: OAuth oauth_consumer_key="[redacted]l", oauth_token="[redacted]", oauth_signature_method="HMAC-SHA1", oauth_version="1.0", oauth_nonce="16710368452795499390", oauth_timestamp="1413938522", oauth_signature="[redacted]"
    Accept-Language: en-us
    Accept-Encoding: gzip, deflate

    Note that since it is a client application it wants to include cookies. I'm not sure if that is contributing to the problem
    I may look into using a different library for Oauth support...

  • little_alien over 6 years ago

    bedhed3000
    Definitely not a problem with the library itself, but rather issues on discogs' end that are making requests using certain libs get rejected. Just guessing. I am trying little_alien's nodejs client to see if I can get it working.

    If you can replicate the problem let me know. I've tried some quick random searches and it seems to work fine. My client has been using the same version of https://www.npmjs.org/package/oauth-1.0a (0.1.1) since its first release, so that shouldn't be a factor.
  • p.sanders over 6 years ago

    I discovered the cause of some of my 401 errors. When using https, I was still signing the reqests as http://..., so that was my fault, sorry about that. But 401 errors at certain (http) customer sites remain, and that's still bothering me.
  • fheidenreich over 6 years ago

    rodneyfool,
    Thanks for looking into this.

    rodneyfool
    In addition to the PM I sent you, it also looks like your search terms are being sent in the OAuth verifier field. Be sure to double check that all values are being sent with the correct keys.

    From what I understand, the oauth_verifier is only sent when requesting an access token and not while accessing protected resources.

    I've also checked the library that I'm using (it's liboauthcpp, the same as p.sanders) and there, the oauth_verifier field is also only send during the phase where the access token is requested.

    You can also see from the requests I've posted above, that no oauth_verifier was sent with those requests so it's also unclear to me how the search terms can go into that field. Can you maybe post the latest of those error messages with the respective request that was sent?

    Thanks!

    Florian
  • invocrown over 6 years ago

    invocrown edited over 6 years ago
    I am having the same problem. I am not experienced with OAuth so it's hard for me to tell if this is my problem or not. But the fact that some calls work and others don't is telling. Here is a test script I wrote:

    <?php
    // This example uses a fork of the OAuthSimple library for PHP
    // found here: https://github.com/tonefolder/oauthsimple/tree/master/php

    require 'lib/oauthsimple/php/OAuthSimple.php';

    function test_discogs_auth($url, $query='')
    {
    $oauth_consumer_key = '...';
    $oauth_consumer_secret = '...';
    $oauth_token = '...';
    $oauth_token_secret = '...';
    $oauthObject = new OAuthSimple($oauth_consumer_key, $oauth_consumer_secret);

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, 'Testing-OAuth/0.1 +http://example.com');

    $result = $oauthObject->sign(Array(
    'path' => $url,
    'signatures'=> Array('oauth_token' => $oauth_token, 'oauth_secret' => $oauth_token_secret)
    ));

    $url = $url . $query;
    curl_setopt($ch, CURLOPT_HTTPHEADER, Array('Authorization: ' . $result['header']));
    curl_setopt($ch, CURLOPT_USERAGENT, 'YOUR_USER_AGENT/0.1 +http://example.com');
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    //Execute the curl session
    $output = curl_exec($ch);
    $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);

    echo "Call: $url\n";
    echo "Return: $status (" . ($status == 200 ? "OK" : "FAIL") . ")\n";
    //echo "Signature: ";
    //print_r($result);
    //echo substr($output, 0, 200) . "\n";
    }

    $base_uri = 'https://api.discogs.com';
    test_discogs_auth("$base_uri/oauth/identity");
    test_discogs_auth("$base_uri/database/search");
    test_discogs_auth("$base_uri/database/search", "?type=release&track=Hurricane&artist=Bob%20Dylan");
    test_discogs_auth("$base_uri/artists/108713/releases");
    test_discogs_auth("$base_uri/artists/108713/releases", "?release_title=nevermind");
    test_discogs_auth("$base_uri/artists/108713/releases", "?foo=bar");
    ?>

    Output is:

    Call: https://api.discogs.com/oauth/identity
    Return: 200 (OK)
    Call: https://api.discogs.com/database/search
    Return: 200 (OK)
    Call: https://api.discogs.com/database/search?type=release&track=Hurricane&artist=Bob%20Dylan
    Return: 401 (FAIL)
    Call: https://api.discogs.com/artists/108713/releases
    Return: 200 (OK)
    Call: https://api.discogs.com/artists/108713/releases?release_title=nevermind
    Return: 200 (OK)
    Call: https://api.discogs.com/artists/108713/releases?foo=bar
    Return: 200 (OK)

    ETA: /releases does not require auth it seems.
    ETA2: Changed &20 to %20 per comments below.
  • little_alien over 6 years ago

    invocrown
    Call: https://api.discogs.com/database/search?type=release&track=Hurricane&artist=Bob&20Dylan

    Shouldn't "Bob&20Dylan" be "Bob+Dylan"? Could you try that? You were probably trying to do add a space via %20.

    Don't know if this is related: http://stackoverflow.com/questions/1634271/url-encoding-the-space-character-or-20
  • invocrown over 6 years ago

    You're right, but both "+" and "%20" (and indeed, omitting the artist) return 401.
  • little_alien over 6 years ago

    invocrown
    You're right, but both "+" and "%20" (and indeed, omitting the artist) return 401.

    Ah ok, well that was the only thing odd I noticed in your code above. Not that it would explain a 401, but it was worth a shot.
  • zonesny over 6 years ago

    little_alien
    I've tried some quick random searches and it seems to work fine. My client has been using the same version of https://www.npmjs.org/package/oauth-1.0a (0.1.1) since its first release, so that shouldn't be a factor.


    Hi little_alien, since you seem to have success with the database search, would you mind please posting your header and/or query string? Likewise, if anyone has successfully performed a database search, could you please post header info to this thread? Maybe it will help shed some light. Thanks!
  • p.sanders over 6 years ago

    zonesny, the query string and headers I posted above work, most of the time. It's the few machines on which they don't work that are the problem for me. Most people pass the OAuth fields as headers, but passing them as part of the query string is also supposed to work.
  • zonesny over 6 years ago

    Okay, this just solved the database search for me! Again, I'm using the PHP tonefolder library.

    I noticed that my database search query had an error in the query string (even in the example posted by me earlier in this thread). There is a second question mark in the query string that shouldn't be there, just before the oauth_consumer_key:

    http://api.discogs.com/database/search?q=The%20Police&type=artist?oauth_consumer_key=[MYKEY]&oauth_nonce=ENRF7&oauth_signature=[MYSIGNATURE]&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1413911901&oauth_token=[MYTOKEN]&oauth_version=1.0

    I had appended a ? to the end of the search URL

    $query = OAUTH_HOST . "/database/search?";

    And, I was passing the following array to OAuthSimple's sign() function:

    $params['path'] = $query; //the query string containing encoded key/value pairs (e.g. 'q', 'type', 'per_page')
    $params['signatures'] = $oauth_props; //$oauth_props contains 'oauth_token' and 'oauth_token_secret' from access request

    However, since I had combined my query parameters with the 'path' key, OAuthSimple was added the second question mark when signing the url.

    Instead, I removed the question mark at the end of the search url

    $query = OAUTH_HOST . "/database/search";

    And passed the encoded query params as a separate 'parameters' key in the array:

    $params['parameters'] = $queryParams;

    The search now works correctly.
  • bedhed3000 over 6 years ago

    That worked for me too. Thanks.
  • little_alien over 6 years ago

    zonesny
    would you mind please posting your header and/or query string?

    No need for this any more I presume?
  • invocrown over 6 years ago

    Thanks zonesy. You've helped me get it working as well, by adding the parameters array for the query variables and using the signed_url from the sign() function.

    Here's a version of my function that works correctly.

    function test_discogs_auth($url, $query='')
    {
    $oauth_consumer_key = '...';
    $oauth_consumer_secret = '...';
    $oauth_token = '...';
    $oauth_token_secret = '...';
    $oauthObject = new OAuthSimple($oauth_consumer_key, $oauth_consumer_secret);

    // Get params from query
    $query = trim($query, '?');
    parse_str($query, $params);

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, 'Testing-OAuth/0.1 +http://example.com');

    // rebuild it with the URL of the resource you want to access and the token/secret
    $result = $oauthObject->sign(Array(
    'path' => $url,
    'signatures'=> Array('oauth_token' => $oauth_token, 'oauth_secret' => $oauth_token_secret),
    'parameters' => $params,
    ));

    $url = $result['signed_url'];
    curl_setopt($ch, CURLOPT_USERAGENT, 'YOUR_USER_AGENT/0.1 +http://example.com');
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    //Execute the curl session
    $output = curl_exec($ch);
    $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);

    curl_close($ch);

    // print the JSON output to the page
    echo "Call: $url\n";
    echo "Return: $status (" . ($status == 200 ? "OK" : "FAIL") . ")\n";
    //echo "Signature: ";
    //print_r($result);
    //echo substr($output, 0, 200) . "\n";
    }

    I think the problem with my earlier code was that I signed the URL without the query string. I would prefer to use the Authorization: header if possible, I wonder how this can be achieved.
  • little_alien over 6 years ago

    invocrown
    I think the problem with my earlier code was that I signed the URL without the query string. I would prefer to use the Authorization: header if possible, I wonder how this can be achieved.

    $oauthObject->sign() returns an array with also a 'header' key. Make an Authorization header string like on 5.4.1 here http://oauth.net/core/1.0a/#auth_header , pass that as an extra header to curl and you're good to go :)

    I wouldn't use the query string for oauth myself. Moving this stuff to the http header greatly reduces the risk of undesired oauth parameter manipulation when building the url/querystring.
  • invocrown over 6 years ago

    Yes see my first message -- that's what I originally tried to do, but I couldn't get it working. I suspect now because I did not set up the parameters to sign() correctly.
  • little_alien over 6 years ago

    invocrown
    Yes see my first message -- that's what I originally tried to do

    Oops, missed that!
  • zonesny over 6 years ago

    Glad my post was helpful to others. It was driving me bonkers. You raise a good security point little_alien about using the header rather than the query string.
  • little_alien over 6 years ago

    zonesny
    You raise a good security point little_alien about using the header rather than the query string.

    My remark wasn't really about security. You can manipulate a header just as easily as a querystring. I was referring to the manipulation of the querystring by ones own code to parse the right API request. Not having the OAuth stuff in your querystring also helps with debugging the requests actually sent to Discogs.
  • zonesny over 6 years ago

    Thanks for clarifying.
  • bedhed3000 over 6 years ago


    invocrown
    Yes see my first message -- that's what I originally tried to do, but I couldn't get it working. I suspect now because I did not set up the parameters to sign() correctly.


    I couldn't get it working either. I might be able to get it working by this weekend. If I do, I'll update the example on gisthub. It is already updated to include a line where we need to add the search parameters to the array that gets passed to the sign() method.
  • p.sanders over 6 years ago

    I finally have a reproducible case of a 401 error accessing the /database/search endpoint.

    The following request fails (error 401):

    GET /database/search?artist=Led&oauth_consumer_key=[...]&oauth_nonce=141415128229&oauth_signature=voUeTOuVeUZITy%2BdsaBgMzDjWJo%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1414151282&oauth_token=[...]&oauth_version=1.0&per_page=100&release_title=Zeppelin&type=release HTTP/1.0
    Accept: */*
    Host: api.discogs.com
    User-Agent: VinylStudio

    And the following request succeeds ('no hits'):

    GET /database/search?artist=Ledd&oauth_consumer_key=[...]&oauth_nonce=141415139229&oauth_signature=LGi6D88jaTCy%2Fx3gP4Wc1OiD%2Bkc%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1414151392&oauth_token=[...]&oauth_version=1.0&per_page=100&release_title=Zeppelin&type=release HTTP/1.0
    Accept: */*
    Host: api.discogs.com
    User-Agent: VinylStudio

    Note that the only difference is that the artist in the second request is passed as Ledd, rather than Led.

    Rodney, if you want to try this for yourself, you can do so as follows:

    1. Get and install a copy of VinylStudio from here: http://www.alpinesoft.co.uk/VinylStudio/download.aspx (Windows or Mac).

    2. Start up VinylStudio and create a new 'collection' (it will prompt you to do this).

    3. In the Record window, create a new album with Artist 'Led' and title 'Zeppelin'.

    4. Go to the Split Tracks window and click on 'Lookup Track Listing'.

    5. Click Lookup. VinylStudio will ask you to authorise the app in your Discogs account.

    6. When it tries to perform the actual search, VinylStudio will again ask you to authorise. This is because Discogs returns a 401 error when the search is performed. Click Cancel.

    7.. Change the artist field to Ledd and click Lookup again. This time it works.

    I hope this is helpful. This problem is driving me mad. I plan to switch over to https and use PLAINTEXT authorisation. This seems to work OK.
  • rodneyfool over 6 years ago

    p.sanders
    I hope this is helpful. This problem is driving me mad. I plan to switch over to https and use PLAINTEXT authorisation. This seems to work OK.

    Yeah, if you're still having this issue, using PLAINTEXT + HTTPS is the best solution for now. I'm still looking into this issue, unfortunately. I will get back to you as soon as possible.

    Thanks and sorry for the headaches!
    Rodney
  • Bruji over 6 years ago

    Bruji edited over 6 years ago
    I can also verify that the API endpoint is now failing for my customers when entering multiple words. The spaces get encoded by CDpedia as + for the URL string and that fails. Removing the + encoding and encoding with %20 solves the issues. But it would be nice to get a fix at the source for users using the older versions of the program.
  • fheidenreich over 6 years ago

    I've now also changed my code to use %20 as word separator instead of + (as described as a workaround in my initial post).

    Though, I'm still curious about the root cause of this problem and would like to see it addressed.

    Kind regards
    Florian
  • MD_PRD over 6 years ago

    I'm getting a 401 even without spaces (just a single word). Strange ...

    rodney
    : any progress to report?
  • p.sanders over 6 years ago

    fheidenreich
    I've now also changed my code to use %20 as word separator instead of + (as described as a workaround in my initial post).

    Though, I'm still curious about the root cause of this problem and would like to see it addressed.

    According to the OAuth specification, %20 is the correct encoding to use:

    http://tools.ietf.org/html/rfc5849#section-3.6
  • little_alien over 6 years ago

    little_alien edited over 6 years ago
    I can confirm the Oauth library I use for my client encodes spaces to %20. In addition to a standard http://www.w3schools.com/jsref/jsref_encodeuricomponent.asp it also explicitly encodes the following characters to their percentage values: ! -> %21, * -> %2A, ' -> %27, ( ->%28, ) -> %29

    Edit: This is just for the OAuth header params of course, not the query string of the GET/POST.
  • fheidenreich over 6 years ago

    p.sanders
    According to the OAuth specification, %20 is the correct encoding to use

    The specification talks about the construction of the OAuth signature in Section 3.6:
    The following percent-encoding method is not defined to replace the existing encoding methods defined by [RFC3986] and [W3C.REC-html40-19980424]. It is used only in the construction of the signature base string and the "Authorization" header field.

    In my understanding, it is not applying to the encoding of general query parameters.
  • rodneyfool over 6 years ago

    [quote=MD_PRD] rodney: any progress to report?[/quote]
    No major finds on this end. I've looked through the error log to see any OAuth-related issues, and most of them have different reasons for causing errors. I have contacted most of those users to let them know how to resolve these issues, or at least, where to start debugging the issue.

    I'll continue to comb through these, and hopefully these issues will get fixed. The %20 encoding issue sounds like there may be a discrepancy between the oauth_signatures (using HMAC-SHA1), so I'm wondering if there's a chance some clients conflict with each other in how they go through that process. If the request-generated oauth_signature doesn't match the oauth_signature generated by the OAuth server, the server will respond with a 401, which is what many users are getting.
  • p.sanders over 6 years ago

    rodneyfool
    The %20 encoding issue sounds like there may be a discrepancy between the oauth_signatures (using HMAC-SHA1), so I'm wondering if there's a chance some clients conflict with each other in how they go through that process.

    The repeatable 401 error I reported above is not dependent on spaces in the search string. That is to say, with the two searches described, neither of which contain a space, one search succeeds and the other fails. Furthermore, I have done searches which do contain a space and they (largely) work.
  • p.sanders over 6 years ago

    fheidenreich
    It [%20 encoding of spaces] is used only in the construction of the signature base string and the "Authorization" header field.
    In my understanding, it is not applying to the encoding of general query parameters.

    You're right, sorry.

  • recordsbymail over 6 years ago

    I too started getting 401 errors on the search endpoint with no changes on my end.

    The fix that zonesny and invocrown posted got me working though.

    Thanks!
  • wefunk over 6 years ago

    Another user checking in... and tearing out hair...
    The problem is the search endpoint

    I've tried with + or %20 (fails either way)
    I've tried with and without SSL (fails either way)
    Image requests don't produce OAUTH errors
    API requests for artist endpoints work fine
    Identity requests work fine
    My OAUTH library is OAuth::Lite::Consumer (perl) with LWP

    GET https://api.discogs.com/database/search?type=artist&q=nirvana HTTP/1.1
    Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
    Authorization: OAuth realm="", oauth_consumer_key="MYCONSUMERKEY", oauth_nonce="761941345432d5bce17d", oauth_signature="1adIPMcdGOj5homWZqfAkGTi9Ys%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1414651385", oauth_token="MYTOKEN", oauth_version="1.0"
    User-Agent: wefunkradio.com/1.0

    HTTP/1.1 401 Unauthorized
    Cache-Control: public, must-revalidate
    Connection: close
    Date: Thu, 30 Oct 2014 06:43:05 GMT
    Server: lighttpd
    WWW-Authenticate: OAuth realm="http://api.discogs.com"
    Content-Length: 61
    Content-Type: application/json
    Access-Control-Allow-Origin: *
    Client-Date: Thu, 30 Oct 2014 06:43:05 GMT
    Client-Peer: 70.103.226.21:443
    Client-Response-Num: 1
    Client-SSL-Cert-Issuer: /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
    Client-SSL-Cert-Subject: /C=US/ST=Oregon/L=Beaverton/O=Discogs, Inc./OU=IT/CN=*.discogs.com
    Client-SSL-Cipher: RC4-SHA
    Client-SSL-Socket-Class: IO::Socket::SSL
    Client-Warning: Unsupported authentication scheme 'oauth'
    Reproxy-Status: yes

    {"message": "You must authenticate to access this resource."}

    GET https://api.discogs.com/oauth/identity HTTP/1.1
    Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
    Authorization: OAuth realm="", oauth_consumer_key="MYCONSUMERKEY", oauth_nonce="ef8a02ea80b22987d306", oauth_signature="VJf5b2VmeYlX1qKHH29%2F6GlHTb0%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1414651542", oauth_token="MYTOKEN", oauth_version="1.0"
    User-Agent: wefunkradio.com/1.0

    HTTP/1.1 200 OK
    Cache-Control: public, must-revalidate
    Connection: close
    Date: Thu, 30 Oct 2014 06:45:42 GMT
    Server: lighttpd
    Content-Length: 125
    Content-Type: application/json
    Access-Control-Allow-Origin: *
    Client-Date: Thu, 30 Oct 2014 06:45:43 GMT
    Client-Peer: 70.103.226.21:443
    Client-Response-Num: 1
    Client-SSL-Cert-Issuer: /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
    Client-SSL-Cert-Subject: /C=US/ST=Oregon/L=Beaverton/O=Discogs, Inc./OU=IT/CN=*.discogs.com
    Client-SSL-Cipher: RC4-SHA
    Client-SSL-Socket-Class: IO::Socket::SSL
    Reproxy-Status: yes

    {"username": "wefunk", "resource_url": "https://api.discogs.com/users/wefunk", "consumer_name": "WEFUNK Radio", "id": 443468}
  • little_alien over 6 years ago

    wefunk
    Another user checking in...

    OAuth realm="" ? I think that should be filled with "https://api.discogs.com" for HTTPS calls.
    http://stackoverflow.com/questions/8577428/what-does-oauth-1-0-realm-stands-for

    Oh and most importantly: Client-Warning: Unsupported authentication scheme 'oauth' ???There definitely seems something wrong on the Discogs side...
  • nanavar over 6 years ago

    Strangely I am also unable to replicate the 401's. I've recently switched my Node.js client library to https after some playing around with it and everything seems to functioning normally. I haven't deployed it in a real production environment yet though..
  • little_alien over 6 years ago

    ^^^ Hey, those are my lines!

    nanavar = (fem)bot/scammer?
  • wefunk over 6 years ago

    little_alien
    OAuth realm="" ? I think that should be filled with "https://api.discogs.com" for HTTPS calls.
    http://stackoverflow.com/questions/8577428/what-does-oauth-1-0-realm-stands-for


    That's an interesting idea. I tried just now including either https://api.discogs.com or http://api.discogs.com as the realm value, rather than blank. It had no effect. I received the same strange 401 response with "Client-Warning: Unsupported authentication scheme 'oauth'".

    I imagine some oauth services might require the realm parameter, but it appears that Discogs doesn't. Identity and non-search endpoints are served fine without a realm value being provided (see my identity request in my earlier post).

    There seems to be something different (and failure prone) about how oauth requests to the search endpoint are handled on the server, compared to other endpoints.

    GET http://api.discogs.com/database/search?type=artist&q=nirvana HTTP/1.1
    Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
    Authorization: OAuth realm="http://api.discogs.com", oauth_consumer_key="MYCONSUMERKEY", oauth_nonce="4de337ad4bc37905910c", oauth_signature="DTiLg9qrIXjZ4DBcgevU58U4jmE%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1414682315", oauth_token="MYOAUTHTOKEN", oauth_version="1.0"
    User-Agent: wefunkradio.com/1.0

    HTTP/1.1 401 Unauthorized
    Cache-Control: public, must-revalidate
    Connection: close
    Date: Thu, 30 Oct 2014 15:18:35 GMT
    Server: lighttpd
    WWW-Authenticate: OAuth realm="http://api.discogs.com"
    Content-Length: 61
    Content-Type: application/json
    Access-Control-Allow-Origin: *
    Client-Date: Thu, 30 Oct 2014 15:18:35 GMT
    Client-Peer: 70.103.226.21:80
    Client-Response-Num: 1
    Client-Warning: Unsupported authentication scheme 'oauth'
    Reproxy-Status: yes

    {"message": "You must authenticate to access this resource."}
  • wefunk over 6 years ago

    At least for the identity endpoint, realm is apparently neither required nor validated:

    GET http://api.discogs.com/oauth/identity HTTP/1.1
    Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
    Authorization: OAuth realm="WTFhttp://api.discogs.comWTF", oauth_consumer_key="MYCOMSUMERKEY", oauth_nonce="6f8688257942a90ce104", oauth_signature="93Bm4R4%2B6Aq5Ra1bFNxBoT49RCQ%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1414682945", oauth_token="MYTOKEN", oauth_version="1.0"
    User-Agent: wefunkradio.com/1.0

    HTTP/1.1 200 OK
    Cache-Control: public, must-revalidate
    Connection: close
    Date: Thu, 30 Oct 2014 15:29:05 GMT
    Server: lighttpd
    Content-Length: 124
    Content-Type: application/json
    Access-Control-Allow-Origin: *
    Client-Date: Thu, 30 Oct 2014 15:29:06 GMT
    Client-Peer: 70.103.226.21:80
    Client-Response-Num: 1
    Reproxy-Status: yes

    {"username": "wefunk", "resource_url": "http://api.discogs.com/users/wefunk", "consumer_name": "WEFUNK Radio", "id": 443468}
  • rodneyfool over 6 years ago

    little_alien
    Oh and most importantly: Client-Warning: Unsupported authentication scheme 'oauth' ???There definitely seems something wrong on the Discogs side...

    I think this warning might be coming from the client library and not the Discogs server. I searched that error message, and a number of results came up in regards to a Perl library.

    As for the search endpoint returning 401s, I'm trying to reproduce it. Have you tried using HTTPS and Plaintext instead of using HMAC-SHA1? Just to potentially isolate the issue.

    Thanks,
    Rodney
  • wefunk over 6 years ago

    rodneyfool
    I think this warning might be coming from the client library and not the Discogs server. I searched that error message, and a number of results came up in regards to a Perl library.


    Ah yes, that's true. What's happening in my perl code is the Oauth::Lite::Consumer library modifies a prepared HTTP request to add the correct OAuth headers. Then the modified request is issued via the LWP (libwwwperl) library. When the 401 response is received containing a WWW-authenticate header, LWP checks to see if it recognizes the authentication method (e.g. Basic, Digest, etc), and since it doesn't know about OAuth itself, it adds the warning headers to the response. It's really just another indication that the OAuth request failed.

    Here's a raw request and response captured via tcpdump:

    GET /database/search?type=artist&q=nirvana HTTP/1.1
    TE: deflate,gzip;q=0.3
    Connection: TE, close
    Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
    Authorization: OAuth realm="http://api.discogs.com", oauth_consumer_key="MYCONSUMERKEY", oauth_nonce="855d6a2d0e54eecad58d", oauth_signature="rVel4NHz3OMNQybmnWUkwlyPiOw%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1414689247", oauth_token="MYOAUTHTOKEN", oauth_version="1.0"
    Host: api.discogs.com
    User-Agent: wefunkradio.com/1.0

    HTTP/1.1 401 Unauthorized
    Reproxy-Status: yes
    Access-Control-Allow-Origin: *
    Cache-Control: public, must-revalidate
    Content-Type: application/json
    WWW-Authenticate: OAuth realm="http://api.discogs.com"
    Content-Length: 61
    Connection: close
    Date: Thu, 30 Oct 2014 17:14:08 GMT
    Server: lighttpd

    {"message": "You must authenticate to access this resource."}

    rodneyfool
    Have you tried using HTTPS and Plaintext instead of using HMAC-SHA1? Just to potentially isolate the issue.


    Thanks, I'll do this and report back

  • p.sanders over 6 years ago

    rodneyfool
    Have you tried using HTTPS and Plaintext instead of using HMAC-SHA1? Just to potentially isolate the issue.

    That works for me, and it's what I intend to do in my next release.
  • wefunk over 6 years ago

    rodneyfool
    Have you tried using HTTPS and Plaintext instead of using HMAC-SHA1? Just to potentially isolate the issue.


    I can confirm - switching to the PLAINTEXT signature method with HTTPS works for me. I'll continue using this method.

    So it seems that identity, artist endpoints, images etc work with HMAC-SHA1 and PLAINTEXT, but database searches only work with PLAINTEXT. Odd.
  • rodneyfool over 6 years ago

    wefunk
    I can confirm - switching to the PLAINTEXT signature method with HTTPS works for me. I'll continue using this method.

    Great, glad it's working. Since we now support HTTPS, I'm considering updating the documentation examples to use HTTPS + PLAINTEXT as it's much more simple to get setup and still secure.
  • little_alien over 6 years ago

    rodneyfool
    I think this warning might be coming from the client library and not the Discogs server. I searched that error message, and a number of results came up in regards to a Perl library.

    Ah ok, my bad. Well, the Client- prefix already gave it away I guess. I assumed that if it was in the response, it would have to come from Discogs...

    rodneyfool
    HTTPS + PLAINTEXT

    So we can conclude it's a difference in HMAC-SHA1 implementation between client and server? Then I still think it's weird the search is the only problem area.
  • jough.dempsey over 6 years ago

    I'm having trouble making an identity request after getting my user's token/secret from the docs' Step 4:

    {"data":{"message":"You must authenticate to access this resource."},"status":401,"config":{"method":"GET","transformRequest":[null],"transformResponse":[null],"headers":{"Authorization":"OAuth oauth_consumer_key=\"[MY CONSUMER KEY]\", oauth_nonce=\"YD3SdSlm0iVl9cJwOzG6bOjGwqgv5ULO\", oauth_signature=\"[MY SECRET]%26[oauth_token_secret]\", oauth_signature_method=\"PLAINTEXT\", oauth_timestamp=\"1428207352\", oauth_token=\"[oauth_token]\", oauth_version=\"1.0a\"","Accept":"application/json, text/plain, */*","User-Agent":"[MY USER AGENT]"},"url":"https://api.discogs.com/oauth/identity"},"statusText":"Unauthorized"}

    I'm sure I'm just doing something wrong.

Log In You must be logged in to post.