java爬取并下载酷狗TOP500歌曲的方法

三叶草

这篇文章主要介绍了java爬取并下载酷狗TOP500歌曲的方法，非常具有实用价值，需要的朋友可以参考下
是这样的，之前买车送的垃圾记录仪不能用了，这两天狠心买了好点的记录仪，带导航、音乐、蓝牙、4g等功能，寻思，既然有这些功能就利用起来，用4g听歌有点奢侈，就准备去酷狗下点歌听，居然都是需要办会员才能下载，而且vip一月只能下载300首，我这么穷又这么抠怎么可能冲会员，于是百度搜了下怎么免费下载，都是python爬取，虽然也会一点，但是电脑上没安装python，再安装再研究感觉有点费劲，于是就花了半小时做了这个爬虫，技术一般，只记录分析实现过程，大牛请绕行。其中用到了一些库，包括：jsoup、httpclient、net.sf.json大家可以自行去下载jar包
1、分析是否能获得top500歌单
首先，打开酷狗首页查看酷狗top500，说好的500首，怎么就只有22首呢，

是真的只让看这些还是能找到其余的呢，于是我就看了下这top500的链接

https://www.kugou.com/yy/rank/home/1-8888.html?from=rank

可以看的出home后边有个1，难道这是代表第一页的意思？于是我就把1改成2，进入，果然进入了第二页，至此可以知道我们可以在网页里获取这500首的歌单。
2.分析找到真正的mp3下载地址（这个有点绕）
点一个歌曲进入播放页面，使用谷歌浏览器的控制台的elements，搜一下mp3，很轻松就定位到了mp3的位置，

但是使用java访问的时候爬取的html里却没有该mp3的文件地址，那么这肯定是在该页面的位置使用了js来加载mp3，那么刷新下网页，看网页加载了哪些东西，加载的东西有点多，着重看一下js、php的请求，主要是看里面有没有mp3的地址，分析细节就不用说了，

最终我在列表的

https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jquery191027067069941080546_1546235744250&hash=667939c6e784265d541deee65ae4f2f8&album_id=0&_=1546235744251

这个请求里发现了mp3的完整地址，

"play_url": "http://fs.w.kugou.com/201812311325/dcf5b6449160903c6ee48035e11434bb/g128/m08/02/09/iicbafrzqf2anoadadn94ubomau995.mp3",

那这个js是怎么判断是哪首歌的呢，那么只可能是hash这个参数来决定歌曲的，然后到播放页面里找到这个hash的位置，是在下面的js里

var datafromsmarty = [{"hash":"667939c6e784265d541deee65ae4f2f8","timelength":"237051","audio_name":"u767du5c0fu767d - u6700u7f8eu5a5au793c","author_name":"u767du5c0fu767d","song_name":"u6700u7f8eu5a5au793c","album_id":0}],//当前页面歌曲信息
    playtype = "search_single";//当前播放
  </script>

在去java爬取该网页，查看能否爬到这个hash，果然，爬取的html里有这段js，到现在mp3的地址也找到了，歌单也找到了，那么下一步就用程序实现就可以了。
3.java实现爬取酷狗mp3
先看一下爬取结果

找到了资源，程序实现就好说了，其中使用到了自己写的几个工具类，自己整理点自己的工具类还是有好处的，以后遇到什么问题就没必要重新写了，直接拿来用就可以了。没什么好说的了，下面直接贴出源码
spiderkugou.java

package com.bing.spider; 
 
import java.io.ioexception; 
import java.util.regex.matcher; 
import java.util.regex.pattern; 
 
import org.jsoup.nodes.document; 
import org.jsoup.nodes.element; 
import org.jsoup.select.elements; 
 
import com.bing.download.filedownload; 
import com.bing.html.htmlmanage; 
import com.bing.http.httpgetconnect; 
 
import net.sf.json.jsonobject; 
 
public class spiderkugou { 
 
  public static string filepath = "f:/music/"; 
  public static string mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jquery191027067069941080546_1546235744250&" 
      + "hash=hash&album_id=0&_=time"; 
   
  public static string link = "https://www.kugou.com/yy/rank/home/page-8888.html?from=rank"; 
  //"https://www.kugou.com/yy/rank/home/page-23784.html?from=rank"; 
   
   
  public static void main(string[] args) throws ioexception { 
     
    for(int i = 1 ; i < 23 ; i++){ 
      string url = link.replace("page", i + ""); 
      gettitle(url); 
      //download("https://www.kugou.com/song/mfy6je5.html"); 
    } 
  } 
   
  public static string gettitle(string url) throws ioexception{ 
    httpgetconnect connect = new httpgetconnect(); 
    string content = connect.connect(url, "utf-8"); 
    htmlmanage html = new htmlmanage(); 
    document doc = html.manage(content); 
    element ele = doc.getelementsbyclass("pc_temp_songlist").get(0); 
    elements eles = ele.getelementsbytag("li"); 
    for(int i = 0 ; i < eles.size() ; i++){ 
      element item = eles.get(i); 
      string title = item.attr("title").trim(); 
      string link = item.getelementsbytag("a").first().attr("href"); 
       
      download(link,title); 
    } 
    return null; 
  } 
   
  public static string download(string url,string name) throws ioexception{ 
    string hash = ""; 
    httpgetconnect connect = new httpgetconnect(); 
    string content = connect.connect(url, "utf-8"); 
    htmlmanage html = new htmlmanage(); 
     
    string regex = ""hash":"[0-9a-z]+""; 
    // 编译正则表达式 
    pattern pattern = pattern.compile(regex); 
    matcher matcher = pattern.matcher(content); 
    if (matcher.find()) { 
      hash = matcher.group(); 
      hash = hash.replace(""hash":"", ""); 
      hash = hash.replace(""", ""); 
    } 
     
    string item = mp3.replace("hash", hash); 
    item = item.replace("time", system.currenttimemillis() + ""); 
 
    system.out.println(item); 
    string mp = connect.connect(item, "utf-8"); 
     
    mp = mp.substring(mp.indexof("(") + 1, mp.length() - 3); 
     
    jsonobject json = jsonobject.fromobject(mp); 
    string playurl = json.getjsonobject("data").getstring("play_url"); 
     
 
    system.out.print(playurl + " == "); 
    filedownload down = new filedownload(); 
    down.download(playurl, filepath + name + ".mp3"); 
     
    system.out.println(name + "下载完成"); 
    return playurl; 
  } 
 
}

httpgetconnect.java

package com.bing.http;
 
import java.io.bufferedreader;
import java.io.ioexception;
import java.io.inputstream;
import java.io.inputstreamreader;
import java.security.nosuchalgorithmexception;
import java.security.cert.certificateexception;
import java.security.cert.x509certificate;
 
import javax.net.ssl.sslcontext;
import javax.net.ssl.trustmanager;
import javax.net.ssl.x509trustmanager;
 
import org.apache.commons.logging.log;
import org.apache.commons.logging.logfactory;
import org.apache.http.httpentity;
import org.apache.http.client.clientprotocolexception;
import org.apache.http.client.httpclient;
import org.apache.http.client.responsehandler;
import org.apache.http.client.config.requestconfig;
import org.apache.http.client.methods.closeablehttpresponse;
import org.apache.http.client.methods.httpget;
import org.apache.http.conn.clientconnectionmanager;
import org.apache.http.conn.scheme.scheme;
import org.apache.http.conn.scheme.schemeregistry;
import org.apache.http.conn.ssl.sslsocketfactory;
import org.apache.http.impl.client.basicresponsehandler;
import org.apache.http.impl.client.closeablehttpclient;
import org.apache.http.impl.client.defaulthttpclient;
import org.apache.http.impl.client.httpclients;
import org.apache.http.impl.conn.basichttpclientconnectionmanager;
import org.apache.http.params.httpparams; 
/**
 * @说明：
 * @author: gaoll
 * @createtime:2014-11-13
 * @modifytime:2014-11-13
 */
public class httpgetconnect {
  
  /**
   * 获取html内容
   * @param url
   * @param charsetname utf-8、gb2312
   * @return
   * @throws ioexception
   */
  public static string connect(string url,string charsetname) throws ioexception{
    basichttpclientconnectionmanager connmanager = new basichttpclientconnectionmanager();
    
    closeablehttpclient httpclient = httpclients.custom()
      .setconnectionmanager(connmanager)
      .build();
    string content = "";
    
    try{
      httpget httpget = new httpget(url);
      
      requestconfig requestconfig = requestconfig.custom()
      .setsockettimeout(5000)
      .setconnecttimeout(50000)
      .setconnectionrequesttimeout(50000)
      .build();
    httpget.setconfig(requestconfig);
    httpget.setheader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    httpget.setheader("accept-encoding", "gzip,deflate,sdch");
    httpget.setheader("accept-language", "zh-cn,zh;q=0.8");
    httpget.setheader("connection", "keep-alive");
    httpget.setheader("upgrade-insecure-requests", "1");
    httpget.setheader("user-agent", "mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome/45.0.2454.101 safari/537.36");
    //httpget.setheader("hosts", "www.oschina.net");
    httpget.setheader("cache-control", "max-age=0"); 
      
      closeablehttpresponse response = httpclient.execute(httpget);
      
      int status = response.getstatusline().getstatuscode();
    if (status >= 200 && status < 300) {
    
    httpentity entity = response.getentity();
    inputstream instream = entity.getcontent();
    bufferedreader br = new bufferedreader(new inputstreamreader(instream,charsetname));
        stringbuffer sbf = new stringbuffer();
        string line = null;
        while ((line = br.readline()) != null){
          sbf.append(line + " ");
        }
 
        br.close();
        content = sbf.tostring();
    } else {
    content = "";
    }
      
    }catch(exception e){
      e.printstacktrace();
    }finally{
      httpclient.close();
    }
    //log.info("content is " + content);
    return content;
  }
  private static log log = logfactory.getlog(httpgetconnect.class);
}

htmlmanage.java

package com.bing.html;
 
import java.io.ioexception;
import java.util.arraylist;
import java.util.list;
 
import org.apache.commons.logging.log;
import org.apache.commons.logging.logfactory;
import org.jsoup.jsoup;
import org.jsoup.nodes.document;
import org.jsoup.nodes.element;
import org.jsoup.select.elements;
 
import com.bing.http.httpgetconnect;
 
/**
 * @说明：
 * @author: gaoll
 * @createtime:2014-11-13
 * @modifytime:2014-11-13
 */
public class htmlmanage {
  
  public document manage(string html){
    document doc = jsoup.parse(html);
    return doc;
  }
  
  public document managedirect(string url) throws ioexception{
    document doc = jsoup.connect( url ).get();
    return doc;
  }
  
  public list<string> managehtmltag(document doc,string tag ){
    list<string> list = new arraylist<string>();
    
    elements elements = doc.getelementsbytag(tag);
    for(int i = 0; i < elements.size() ; i++){
      string str = elements.get(i).html();
      list.add(str);
    }
    return list;
  }
  
  public list<string> managehtmlclass(document doc,string clas ){
    list<string> list = new arraylist<string>();
    
    elements elements = doc.getelementsbyclass(clas);
    for(int i = 0; i < elements.size() ; i++){
      string str = elements.get(i).html();
      list.add(str);
    }
    return list;
  }
  
  public list<string> managehtmlkey(document doc,string key,string value ){
    list<string> list = new arraylist<string>();
    
    elements elements = doc.getelementsbyattributevalue(key, value);
    for(int i = 0; i < elements.size() ; i++){
      string str = elements.get(i).html();
      list.add(str);
    }
    return list;
  }
  
  private static log log = logfactory.getlog(htmlmanage.class);
}

filedownload.java

package com.bing.download;
 
import java.io.bufferedinputstream;
import java.io.bufferedoutputstream;
import java.io.file;
import java.io.fileoutputstream;
 
import org.apache.commons.logging.log;
import org.apache.commons.logging.logfactory;
import org.apache.http.client.config.requestconfig;
import org.apache.http.client.methods.closeablehttpresponse;
import org.apache.http.client.methods.httpget;
import org.apache.http.impl.client.closeablehttpclient;
import org.apache.http.impl.client.httpclients;
 
/**
 * @说明：
 * @author: gaoll
 * @createtime:2014-11-20
 * @modifytime:2014-11-20
 */
public class filedownload {
  
  /**
   * 文件下载
   * @param url 链接地址
   * @param path 要保存的路径及文件名
   * @return
   */
  public static boolean download(string url,string path){
    
    boolean flag = false;
    
    closeablehttpclient httpclient = httpclients.createdefault();
    requestconfig requestconfig = requestconfig.custom().setsockettimeout(2000)
        .setconnecttimeout(2000).build();
 
    httpget get = new httpget(url);
    get.setconfig(requestconfig);
    
    bufferedinputstream in = null;
    bufferedoutputstream out = null;
    try{
      for(int i=0;i<3;i++){
        closeablehttpresponse result = httpclient.execute(get);
        system.out.println(result.getstatusline());
        if(result.getstatusline().getstatuscode() == 200){
          in = new bufferedinputstream(result.getentity().getcontent());
          file file = new file(path);
          out = new bufferedoutputstream(new fileoutputstream(file));
          byte[] buffer = new byte[1024];
          int len = -1;
          while((len = in.read(buffer,0,1024)) > -1){
            out.write(buffer,0,len);
          }
          flag = true;
          break;
        }else if(result.getstatusline().getstatuscode() == 500){
          continue ;
        }
      }
      
    }catch(exception e){
      e.printstacktrace();
      flag = false;
    }finally{
      get.releaseconnection();
      try{
        if(in != null){
          in.close();
        }
        if(out != null){
          out.close();
        }
      }catch(exception e){
        e.printstacktrace();
        flag = false;
      }
    }
    return flag;
  }
 
  private static log log = logfactory.getlog(filedownload.class);
}

到这就结束了，有可能有些代码没贴全，主要代码已经差不多，应该可以跑起来，多多指教。
以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持CodeAE代码之家。
原文链接：https://my.oschina.net/gllfeixiang/blog/2995570

[Java] java爬取并下载酷狗TOP500歌曲的方法

相关帖子

常用JavaScript的注释方法示例

怎么在JavaScript中加注释

常用HTML网页跳转的几种方法

Win10取消开机密码的2种方法

js保留两位小数方法总结

php使用GD库创建图片缩略图的方法

Discuz应用中心恢复授权的方法

Discuz X重置应用中心数据的方法

react如何引入css

阿里云盘获取refresh_token方法

三叶草 LV3