mirror of
https://codeberg.org/polarisfm/youtube-dl
synced 2025-01-07 13:47:54 +01:00
Merge branch 'master' of https://github.com/rg3/youtube-dl
This commit is contained in:
J. Randall Owens
commit
edc7e70d9f
6
.github/ISSUE_TEMPLATE/1_broken_site.md
vendored
6
.github/ISSUE_TEMPLATE/1_broken_site.md
vendored
@ -18,7 +18,7 @@ title: ''
|
||||
|
||||
<!--
|
||||
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.02. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.30. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
|
||||
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
|
||||
- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
|
||||
@ -26,7 +26,7 @@ Carefully read and work through this check list in order to prevent the most com
|
||||
-->
|
||||
|
||||
- [ ] I'm reporting a broken site support
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.02**
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.30**
|
||||
- [ ] I've checked that all provided URLs are alive and playable in a browser
|
||||
- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
|
||||
- [ ] I've searched the bugtracker for similar issues including closed ones
|
||||
@ -41,7 +41,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
|
||||
[debug] User config: []
|
||||
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
|
||||
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
|
||||
[debug] youtube-dl version 2019.07.02
|
||||
[debug] youtube-dl version 2019.07.30
|
||||
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
|
||||
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
|
||||
[debug] Proxy map: {}
|
||||
|
||||
@ -19,7 +19,7 @@ labels: 'site-support-request'
|
||||
|
||||
<!--
|
||||
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.02. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.30. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
|
||||
- Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
|
||||
- Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
|
||||
@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
|
||||
-->
|
||||
|
||||
- [ ] I'm reporting a new site support request
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.02**
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.30**
|
||||
- [ ] I've checked that all provided URLs are alive and playable in a browser
|
||||
- [ ] I've checked that none of provided URLs violate any copyrights
|
||||
- [ ] I've searched the bugtracker for similar site support requests including closed ones
|
||||
|
||||
@ -18,13 +18,13 @@ title: ''
|
||||
|
||||
<!--
|
||||
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.02. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.30. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
|
||||
- Finally, put x into all relevant boxes (like this [x])
|
||||
-->
|
||||
|
||||
- [ ] I'm reporting a site feature request
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.02**
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.30**
|
||||
- [ ] I've searched the bugtracker for similar site feature requests including closed ones
|
||||
|
||||
|
||||
|
||||
6
.github/ISSUE_TEMPLATE/4_bug_report.md
vendored
6
.github/ISSUE_TEMPLATE/4_bug_report.md
vendored
@ -18,7 +18,7 @@ title: ''
|
||||
|
||||
<!--
|
||||
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.02. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.30. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
|
||||
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
|
||||
- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
|
||||
@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
|
||||
-->
|
||||
|
||||
- [ ] I'm reporting a broken site support issue
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.02**
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.30**
|
||||
- [ ] I've checked that all provided URLs are alive and playable in a browser
|
||||
- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
|
||||
- [ ] I've searched the bugtracker for similar bug reports including closed ones
|
||||
@ -43,7 +43,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
|
||||
[debug] User config: []
|
||||
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
|
||||
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
|
||||
[debug] youtube-dl version 2019.07.02
|
||||
[debug] youtube-dl version 2019.07.30
|
||||
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
|
||||
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
|
||||
[debug] Proxy map: {}
|
||||
|
||||
4
.github/ISSUE_TEMPLATE/5_feature_request.md
vendored
4
.github/ISSUE_TEMPLATE/5_feature_request.md
vendored
@ -19,13 +19,13 @@ labels: 'request'
|
||||
|
||||
<!--
|
||||
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.02. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2019.07.30. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
|
||||
- Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
|
||||
- Finally, put x into all relevant boxes (like this [x])
|
||||
-->
|
||||
|
||||
- [ ] I'm reporting a feature request
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.02**
|
||||
- [ ] I've verified that I'm running youtube-dl version **2019.07.30**
|
||||
- [ ] I've searched the bugtracker for similar feature requests including closed ones
|
||||
|
||||
|
||||
|
||||
81
ChangeLog
81
ChangeLog
@ -1,3 +1,84 @@
|
||||
version 2019.07.30
|
||||
|
||||
Extractors
|
||||
* [youtube] Fix and improve title and description extraction (#21934)
|
||||
|
||||
|
||||
version 2019.07.27
|
||||
|
||||
Extractors
|
||||
+ [yahoo:japannews] Add support for yahoo.co.jp (#21698, #21265)
|
||||
+ [discovery] Add support go.discovery.com URLs
|
||||
* [youtube:playlist] Relax video regular expression (#21844)
|
||||
* [generic] Restrict --default-search schemeless URLs detection pattern
|
||||
(#21842)
|
||||
* [vrv] Fix CMS signing query extraction (#21809)
|
||||
|
||||
|
||||
version 2019.07.16
|
||||
|
||||
Extractors
|
||||
+ [asiancrush] Add support for yuyutv.com, midnightpulp.com and cocoro.tv
|
||||
(#21281, #21290)
|
||||
* [kaltura] Check source format URL (#21290)
|
||||
* [ctsnews] Fix YouTube embeds extraction (#21678)
|
||||
+ [einthusan] Add support for einthusan.com (#21748, #21775)
|
||||
+ [youtube] Add support for invidious.mastodon.host (#21777)
|
||||
+ [gfycat] Extend URL regular expression (#21779, #21780)
|
||||
* [youtube] Restrict is_live extraction (#21782)
|
||||
|
||||
|
||||
version 2019.07.14
|
||||
|
||||
Extractors
|
||||
* [porn91] Fix extraction (#21312)
|
||||
+ [yandexmusic] Extract track number and disk number (#21421)
|
||||
+ [yandexmusic] Add support for multi disk albums (#21420, #21421)
|
||||
* [lynda] Handle missing subtitles (#20490, #20513)
|
||||
+ [youtube] Add more invidious instances to URL regular expression (#21694)
|
||||
* [twitter] Improve uploader id extraction (#21705)
|
||||
* [spankbang] Fix and improve metadata extraction
|
||||
* [spankbang] Fix extraction (#21763, #21764)
|
||||
+ [dlive] Add support for dlive.tv (#18080)
|
||||
+ [livejournal] Add support for livejournal.com (#21526)
|
||||
* [roosterteeth] Fix free episode extraction (#16094)
|
||||
* [dbtv] Fix extraction
|
||||
* [bellator] Fix extraction
|
||||
- [rudo] Remove extractor (#18430, #18474)
|
||||
* [facebook] Fallback to twitter:image meta for thumbnail extraction (#21224)
|
||||
* [bleacherreport] Fix Bleacher Report CMS extraction
|
||||
* [espn] Fix fivethirtyeight.com extraction
|
||||
* [5tv] Relax video URL regular expression and support https URLs
|
||||
* [youtube] Fix is_live extraction (#21734)
|
||||
* [youtube] Fix authentication (#11270)
|
||||
|
||||
|
||||
version 2019.07.12
|
||||
|
||||
Core
|
||||
+ [adobepass] Add support for AT&T U-verse (mso ATT) (#13938, #21016)
|
||||
|
||||
Extractors
|
||||
+ [mgtv] Pass Referer HTTP header for format URLs (#21726)
|
||||
+ [beeg] Add support for api/v6 v2 URLs without t argument (#21701)
|
||||
* [voxmedia:volume] Improvevox embed extraction (#16846)
|
||||
* [funnyordie] Move extraction to VoxMedia extractor (#16846)
|
||||
* [gameinformer] Fix extraction (#8895, #15363, #17206)
|
||||
* [funk] Fix extraction (#17915)
|
||||
* [packtpub] Relax lesson URL regular expression (#21695)
|
||||
* [packtpub] Fix extraction (#21268)
|
||||
* [philharmoniedeparis] Relax URL regular expression (#21672)
|
||||
* [peertube] Detect embed URLs in generic extraction (#21666)
|
||||
* [mixer:vod] Relax URL regular expression (#21657, #21658)
|
||||
+ [lecturio] Add support id based URLs (#21630)
|
||||
+ [go] Add site info for disneynow (#21613)
|
||||
* [ted] Restrict info regular expression (#21631)
|
||||
* [twitch:vod] Actualize m3u8 URL (#21538, #21607)
|
||||
* [vzaar] Fix videos with empty title (#21606)
|
||||
* [tvland] Fix extraction (#21384)
|
||||
* [arte] Clean extractor (#15583, #21614)
|
||||
|
||||
|
||||
version 2019.07.02
|
||||
|
||||
Core
|
||||
|
||||
@ -58,16 +58,8 @@
|
||||
- **ARD:mediathek**
|
||||
- **ARDBetaMediathek**
|
||||
- **Arkena**
|
||||
- **arte.tv**
|
||||
- **arte.tv:+7**
|
||||
- **arte.tv:cinema**
|
||||
- **arte.tv:concert**
|
||||
- **arte.tv:creative**
|
||||
- **arte.tv:ddc**
|
||||
- **arte.tv:embed**
|
||||
- **arte.tv:future**
|
||||
- **arte.tv:info**
|
||||
- **arte.tv:magazine**
|
||||
- **arte.tv:playlist**
|
||||
- **AsianCrush**
|
||||
- **AsianCrushPlaylist**
|
||||
@ -231,6 +223,8 @@
|
||||
- **DiscoveryNetworksDe**
|
||||
- **DiscoveryVR**
|
||||
- **Disney**
|
||||
- **dlive:stream**
|
||||
- **dlive:vod**
|
||||
- **Dotsub**
|
||||
- **DouyuShow**
|
||||
- **DouyuTV**: 斗鱼
|
||||
@ -313,9 +307,7 @@
|
||||
- **FrontendMastersCourse**
|
||||
- **FrontendMastersLesson**
|
||||
- **Funimation**
|
||||
- **FunkChannel**
|
||||
- **FunkMix**
|
||||
- **FunnyOrDie**
|
||||
- **Funk**
|
||||
- **Fusion**
|
||||
- **Fux**
|
||||
- **FXNetworks**
|
||||
@ -458,6 +450,7 @@
|
||||
- **linkedin:learning:course**
|
||||
- **LinuxAcademy**
|
||||
- **LiTV**
|
||||
- **LiveJournal**
|
||||
- **LiveLeak**
|
||||
- **LiveLeakEmbed**
|
||||
- **livestream**
|
||||
@ -764,7 +757,6 @@
|
||||
- **rtve.es:television**
|
||||
- **RTVNH**
|
||||
- **RTVS**
|
||||
- **Rudo**
|
||||
- **RUHD**
|
||||
- **rutube**: Rutube videos
|
||||
- **rutube:channel**: Rutube channels
|
||||
@ -896,7 +888,6 @@
|
||||
- **TF1**
|
||||
- **TFO**
|
||||
- **TheIntercept**
|
||||
- **theoperaplatform**
|
||||
- **ThePlatform**
|
||||
- **ThePlatformFeed**
|
||||
- **TheScene**
|
||||
@ -1126,6 +1117,7 @@
|
||||
- **Yahoo**: Yahoo screen and movies
|
||||
- **yahoo:gyao**
|
||||
- **yahoo:gyao:player**
|
||||
- **yahoo:japannews**: Yahoo! Japan News
|
||||
- **YandexDisk**
|
||||
- **yandexmusic:album**: Яндекс.Музыка - Альбом
|
||||
- **yandexmusic:playlist**: Яндекс.Музыка - Плейлист
|
||||
|
||||
@ -53,7 +53,7 @@ class DashSegmentsFD(FragmentFD):
|
||||
except compat_urllib_error.HTTPError as err:
|
||||
# YouTube may often return 404 HTTP error for a fragment causing the
|
||||
# whole download to fail. However if the same fragment is immediately
|
||||
# retried with the same request data this usually succeeds (1-2 attemps
|
||||
# retried with the same request data this usually succeeds (1-2 attempts
|
||||
# is usually enough) thus allowing to download the whole file successfully.
|
||||
# To be future-proof we will retry all fragments that fail with any
|
||||
# HTTP error.
|
||||
|
||||
@ -146,7 +146,7 @@ def write_piff_header(stream, params):
|
||||
sps, pps = codec_private_data.split(u32.pack(1))[1:]
|
||||
avcc_payload = u8.pack(1) # configuration version
|
||||
avcc_payload += sps[1:4] # avc profile indication + profile compatibility + avc level indication
|
||||
avcc_payload += u8.pack(0xfc | (params.get('nal_unit_length_field', 4) - 1)) # complete represenation (1) + reserved (11111) + length size minus one
|
||||
avcc_payload += u8.pack(0xfc | (params.get('nal_unit_length_field', 4) - 1)) # complete representation (1) + reserved (11111) + length size minus one
|
||||
avcc_payload += u8.pack(1) # reserved (0) + number of sps (0000001)
|
||||
avcc_payload += u16.pack(len(sps))
|
||||
avcc_payload += sps
|
||||
|
||||
@ -15,10 +15,13 @@ class AbcNewsVideoIE(AMPIE):
|
||||
IE_NAME = 'abcnews:video'
|
||||
_VALID_URL = r'''(?x)
|
||||
https?://
|
||||
abcnews\.go\.com/
|
||||
(?:
|
||||
[^/]+/video/(?P<display_id>[0-9a-z-]+)-|
|
||||
video/embed\?.*?\bid=
|
||||
abcnews\.go\.com/
|
||||
(?:
|
||||
[^/]+/video/(?P<display_id>[0-9a-z-]+)-|
|
||||
video/embed\?.*?\bid=
|
||||
)|
|
||||
fivethirtyeight\.abcnews\.go\.com/video/embed/\d+/
|
||||
)
|
||||
(?P<id>\d+)
|
||||
'''
|
||||
|
||||
@ -5,14 +5,12 @@ import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from .kaltura import KalturaIE
|
||||
from ..utils import (
|
||||
extract_attributes,
|
||||
remove_end,
|
||||
)
|
||||
from ..utils import extract_attributes
|
||||
|
||||
|
||||
class AsianCrushIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?asiancrush\.com/video/(?:[^/]+/)?0+(?P<id>\d+)v\b'
|
||||
_VALID_URL_BASE = r'https?://(?:www\.)?(?P<host>(?:(?:asiancrush|yuyutv|midnightpulp)\.com|cocoro\.tv))'
|
||||
_VALID_URL = r'%s/video/(?:[^/]+/)?0+(?P<id>\d+)v\b' % _VALID_URL_BASE
|
||||
_TESTS = [{
|
||||
'url': 'https://www.asiancrush.com/video/012869v/women-who-flirt/',
|
||||
'md5': 'c3b740e48d0ba002a42c0b72857beae6',
|
||||
@ -20,7 +18,7 @@ class AsianCrushIE(InfoExtractor):
|
||||
'id': '1_y4tmjm5r',
|
||||
'ext': 'mp4',
|
||||
'title': 'Women Who Flirt',
|
||||
'description': 'md5:3db14e9186197857e7063522cb89a805',
|
||||
'description': 'md5:7e986615808bcfb11756eb503a751487',
|
||||
'timestamp': 1496936429,
|
||||
'upload_date': '20170608',
|
||||
'uploader_id': 'craig@crifkin.com',
|
||||
@ -28,10 +26,27 @@ class AsianCrushIE(InfoExtractor):
|
||||
}, {
|
||||
'url': 'https://www.asiancrush.com/video/she-was-pretty/011886v-pretty-episode-3/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.yuyutv.com/video/013886v/the-act-of-killing/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.yuyutv.com/video/peep-show/013922v-warring-factions/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.midnightpulp.com/video/010400v/drifters/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.midnightpulp.com/video/mononoke/016378v-zashikiwarashi-part-1/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.cocoro.tv/video/the-wonderful-wizard-of-oz/008878v-the-wonderful-wizard-of-oz-ep01/',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
host = mobj.group('host')
|
||||
video_id = mobj.group('id')
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
@ -51,7 +66,7 @@ class AsianCrushIE(InfoExtractor):
|
||||
r'\bentry_id["\']\s*:\s*["\'](\d+)', webpage, 'entry id')
|
||||
|
||||
player = self._download_webpage(
|
||||
'https://api.asiancrush.com/embeddedVideoPlayer', video_id,
|
||||
'https://api.%s/embeddedVideoPlayer' % host, video_id,
|
||||
query={'id': entry_id})
|
||||
|
||||
kaltura_id = self._search_regex(
|
||||
@ -63,15 +78,23 @@ class AsianCrushIE(InfoExtractor):
|
||||
r'/p(?:artner_id)?/(\d+)', player, 'partner id',
|
||||
default='513551')
|
||||
|
||||
return self.url_result(
|
||||
'kaltura:%s:%s' % (partner_id, kaltura_id),
|
||||
ie=KalturaIE.ie_key(), video_id=kaltura_id,
|
||||
video_title=title)
|
||||
description = self._html_search_regex(
|
||||
r'(?s)<div[^>]+\bclass=["\']description["\'][^>]*>(.+?)</div>',
|
||||
webpage, 'description', fatal=False)
|
||||
|
||||
return {
|
||||
'_type': 'url_transparent',
|
||||
'url': 'kaltura:%s:%s' % (partner_id, kaltura_id),
|
||||
'ie_key': KalturaIE.ie_key(),
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'description': description,
|
||||
}
|
||||
|
||||
|
||||
class AsianCrushPlaylistIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?asiancrush\.com/series/0+(?P<id>\d+)s\b'
|
||||
_TEST = {
|
||||
_VALID_URL = r'%s/series/0+(?P<id>\d+)s\b' % AsianCrushIE._VALID_URL_BASE
|
||||
_TESTS = [{
|
||||
'url': 'https://www.asiancrush.com/series/012481s/scholar-walks-night/',
|
||||
'info_dict': {
|
||||
'id': '12481',
|
||||
@ -79,7 +102,16 @@ class AsianCrushPlaylistIE(InfoExtractor):
|
||||
'description': 'md5:7addd7c5132a09fd4741152d96cce886',
|
||||
},
|
||||
'playlist_count': 20,
|
||||
}
|
||||
}, {
|
||||
'url': 'https://www.yuyutv.com/series/013920s/peep-show/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.midnightpulp.com/series/016375s/mononoke/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://www.cocoro.tv/series/008549s/the-wonderful-wizard-of-oz/',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
playlist_id = self._match_id(url)
|
||||
@ -96,15 +128,15 @@ class AsianCrushPlaylistIE(InfoExtractor):
|
||||
entries.append(self.url_result(
|
||||
mobj.group('url'), ie=AsianCrushIE.ie_key()))
|
||||
|
||||
title = remove_end(
|
||||
self._html_search_regex(
|
||||
r'(?s)<h1\b[^>]\bid=["\']movieTitle[^>]+>(.+?)</h1>', webpage,
|
||||
'title', default=None) or self._og_search_title(
|
||||
webpage, default=None) or self._html_search_meta(
|
||||
'twitter:title', webpage, 'title',
|
||||
default=None) or self._search_regex(
|
||||
r'<title>([^<]+)</title>', webpage, 'title', fatal=False),
|
||||
' | AsianCrush')
|
||||
title = self._html_search_regex(
|
||||
r'(?s)<h1\b[^>]\bid=["\']movieTitle[^>]+>(.+?)</h1>', webpage,
|
||||
'title', default=None) or self._og_search_title(
|
||||
webpage, default=None) or self._html_search_meta(
|
||||
'twitter:title', webpage, 'title',
|
||||
default=None) or self._search_regex(
|
||||
r'<title>([^<]+)</title>', webpage, 'title', fatal=False)
|
||||
if title:
|
||||
title = re.sub(r'\s*\|\s*.+?$', '', title)
|
||||
|
||||
description = self._og_search_description(
|
||||
webpage, default=None) or self._html_search_meta(
|
||||
|
||||
@ -32,6 +32,10 @@ class BeegIE(InfoExtractor):
|
||||
# api/v6 v2
|
||||
'url': 'https://beeg.com/1941093077?t=911-1391',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
# api/v6 v2 w/o t
|
||||
'url': 'https://beeg.com/1277207756',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://beeg.porn/video/5416503',
|
||||
'only_matching': True,
|
||||
@ -49,14 +53,17 @@ class BeegIE(InfoExtractor):
|
||||
r'beeg_version\s*=\s*([\da-zA-Z_-]+)', webpage, 'beeg version',
|
||||
default='1546225636701')
|
||||
|
||||
qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
|
||||
t = qs.get('t', [''])[0].split('-')
|
||||
if len(t) > 1:
|
||||
if len(video_id) >= 10:
|
||||
query = {
|
||||
'v': 2,
|
||||
's': t[0],
|
||||
'e': t[1],
|
||||
}
|
||||
qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
|
||||
t = qs.get('t', [''])[0].split('-')
|
||||
if len(t) > 1:
|
||||
query.update({
|
||||
's': t[0],
|
||||
'e': t[1],
|
||||
})
|
||||
else:
|
||||
query = {'v': 1}
|
||||
|
||||
|
||||
@ -6,7 +6,6 @@ from ..utils import (
|
||||
ExtractorError,
|
||||
remove_end,
|
||||
)
|
||||
from .rudo import RudoIE
|
||||
|
||||
|
||||
class BioBioChileTVIE(InfoExtractor):
|
||||
@ -41,11 +40,15 @@ class BioBioChileTVIE(InfoExtractor):
|
||||
}, {
|
||||
'url': 'http://www.biobiochile.cl/noticias/bbtv/comentarios-bio-bio/2016/07/08/edecanes-del-congreso-figuras-decorativas-que-le-cuestan-muy-caro-a-los-chilenos.shtml',
|
||||
'info_dict': {
|
||||
'id': 'edecanes-del-congreso-figuras-decorativas-que-le-cuestan-muy-caro-a-los-chilenos',
|
||||
'id': 'b4xd0LK3SK',
|
||||
'ext': 'mp4',
|
||||
'uploader': '(none)',
|
||||
'upload_date': '20160708',
|
||||
'title': 'Edecanes del Congreso: Figuras decorativas que le cuestan muy caro a los chilenos',
|
||||
# TODO: fix url_transparent information overriding
|
||||
# 'uploader': 'Juan Pablo Echenique',
|
||||
'title': 'Comentario Oscar Cáceres',
|
||||
},
|
||||
'params': {
|
||||
# empty m3u8 manifest
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://tv.biobiochile.cl/notas/2015/10/22/ninos-transexuales-de-quien-es-la-decision.shtml',
|
||||
@ -60,7 +63,9 @@ class BioBioChileTVIE(InfoExtractor):
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
rudo_url = RudoIE._extract_url(webpage)
|
||||
rudo_url = self._search_regex(
|
||||
r'<iframe[^>]+src=(?P<q1>[\'"])(?P<url>(?:https?:)?//rudo\.video/vod/[0-9a-zA-Z]+)(?P=q1)',
|
||||
webpage, 'embed URL', None, group='url')
|
||||
if not rudo_url:
|
||||
raise ExtractorError('No videos found')
|
||||
|
||||
@ -68,7 +73,7 @@ class BioBioChileTVIE(InfoExtractor):
|
||||
|
||||
thumbnail = self._og_search_thumbnail(webpage)
|
||||
uploader = self._html_search_regex(
|
||||
r'<a[^>]+href=["\']https?://(?:busca|www)\.biobiochile\.cl/(?:lista/)?(?:author|autor)[^>]+>(.+?)</a>',
|
||||
r'<a[^>]+href=["\'](?:https?://(?:busca|www)\.biobiochile\.cl)?/(?:lista/)?(?:author|autor)[^>]+>(.+?)</a>',
|
||||
webpage, 'uploader', fatal=False)
|
||||
|
||||
return {
|
||||
|
||||
@ -71,7 +71,7 @@ class BleacherReportIE(InfoExtractor):
|
||||
video = article_data.get('video')
|
||||
if video:
|
||||
video_type = video['type']
|
||||
if video_type == 'cms.bleacherreport.com':
|
||||
if video_type in ('cms.bleacherreport.com', 'vid.bleacherreport.com'):
|
||||
info['url'] = 'http://bleacherreport.com/video_embed?id=%s' % video['id']
|
||||
elif video_type == 'ooyala.com':
|
||||
info['url'] = 'ooyala:%s' % video['id']
|
||||
@ -87,9 +87,9 @@ class BleacherReportIE(InfoExtractor):
|
||||
|
||||
|
||||
class BleacherReportCMSIE(AMPIE):
|
||||
_VALID_URL = r'https?://(?:www\.)?bleacherreport\.com/video_embed\?id=(?P<id>[0-9a-f-]{36})'
|
||||
_VALID_URL = r'https?://(?:www\.)?bleacherreport\.com/video_embed\?id=(?P<id>[0-9a-f-]{36}|\d{5})'
|
||||
_TESTS = [{
|
||||
'url': 'http://bleacherreport.com/video_embed?id=8fd44c2f-3dc5-4821-9118-2c825a98c0e1',
|
||||
'url': 'http://bleacherreport.com/video_embed?id=8fd44c2f-3dc5-4821-9118-2c825a98c0e1&library=video-cms',
|
||||
'md5': '2e4b0a997f9228ffa31fada5c53d1ed1',
|
||||
'info_dict': {
|
||||
'id': '8fd44c2f-3dc5-4821-9118-2c825a98c0e1',
|
||||
@ -101,6 +101,6 @@ class BleacherReportCMSIE(AMPIE):
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
info = self._extract_feed_info('http://cms.bleacherreport.com/media/items/%s/akamai.json' % video_id)
|
||||
info = self._extract_feed_info('http://vid.bleacherreport.com/videos/%s.akamai' % video_id)
|
||||
info['id'] = video_id
|
||||
return info
|
||||
|
||||
@ -220,7 +220,7 @@ class InfoExtractor(object):
|
||||
* "preference" (optional, int) - quality of the image
|
||||
* "width" (optional, int)
|
||||
* "height" (optional, int)
|
||||
* "resolution" (optional, string "{width}x{height"},
|
||||
* "resolution" (optional, string "{width}x{height}",
|
||||
deprecated)
|
||||
* "filesize" (optional, int)
|
||||
thumbnail: Full URL to a video thumbnail image.
|
||||
|
||||
@ -3,6 +3,7 @@ from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import unified_timestamp
|
||||
from .youtube import YoutubeIE
|
||||
|
||||
|
||||
class CtsNewsIE(InfoExtractor):
|
||||
@ -14,8 +15,8 @@ class CtsNewsIE(InfoExtractor):
|
||||
'info_dict': {
|
||||
'id': '201501291578109',
|
||||
'ext': 'mp4',
|
||||
'title': '以色列.真主黨交火 3人死亡',
|
||||
'description': '以色列和黎巴嫩真主黨，爆發五年最嚴重衝突，雙方砲轟交火，兩名以軍死亡，還有一名西班牙籍的聯合國維和人...',
|
||||
'title': '以色列.真主黨交火 3人死亡 - 華視新聞網',
|
||||
'description': '以色列和黎巴嫩真主黨，爆發五年最嚴重衝突，雙方砲轟交火，兩名以軍死亡，還有一名西班牙籍的聯合國維和人員也不幸罹難。大陸陝西、河南、安徽、江蘇和湖北五個省份出現大暴雪，嚴重影響陸空交通，不過九華山卻出現...',
|
||||
'timestamp': 1422528540,
|
||||
'upload_date': '20150129',
|
||||
}
|
||||
@ -26,7 +27,7 @@ class CtsNewsIE(InfoExtractor):
|
||||
'info_dict': {
|
||||
'id': '201309031304098',
|
||||
'ext': 'mp4',
|
||||
'title': '韓國31歲童顏男 貌如十多歲小孩',
|
||||
'title': '韓國31歲童顏男 貌如十多歲小孩 - 華視新聞網',
|
||||
'description': '越有年紀的人，越希望看起來年輕一點，而南韓卻有一位31歲的男子，看起來像是11、12歲的小孩，身...',
|
||||
'thumbnail': r're:^https?://.*\.jpg$',
|
||||
'timestamp': 1378205880,
|
||||
@ -62,8 +63,7 @@ class CtsNewsIE(InfoExtractor):
|
||||
video_url = mp4_feed['source_url']
|
||||
else:
|
||||
self.to_screen('Not CTSPlayer video, trying Youtube...')
|
||||
youtube_url = self._search_regex(
|
||||
r'src="(//www\.youtube\.com/embed/[^"]+)"', page, 'youtube url')
|
||||
youtube_url = YoutubeIE._extract_url(page)
|
||||
|
||||
return self.url_result(youtube_url, ie='Youtube')
|
||||
|
||||
|
||||
@ -7,50 +7,51 @@ from .common import InfoExtractor
|
||||
|
||||
|
||||
class DBTVIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?dbtv\.no/(?:[^/]+/)?(?P<id>[0-9]+)(?:#(?P<display_id>.+))?'
|
||||
_VALID_URL = r'https?://(?:www\.)?dagbladet\.no/video/(?:(?:embed|(?P<display_id>[^/]+))/)?(?P<id>[0-9A-Za-z_-]{11}|[a-zA-Z0-9]{8})'
|
||||
_TESTS = [{
|
||||
'url': 'http://dbtv.no/3649835190001#Skulle_teste_ut_fornøyelsespark,_men_kollegaen_var_bare_opptatt_av_bikinikroppen',
|
||||
'md5': '2e24f67936517b143a234b4cadf792ec',
|
||||
'url': 'https://www.dagbladet.no/video/PynxJnNWChE/',
|
||||
'md5': 'b8f850ba1860adbda668d367f9b77699',
|
||||
'info_dict': {
|
||||
'id': '3649835190001',
|
||||
'display_id': 'Skulle_teste_ut_fornøyelsespark,_men_kollegaen_var_bare_opptatt_av_bikinikroppen',
|
||||
'id': 'PynxJnNWChE',
|
||||
'ext': 'mp4',
|
||||
'title': 'Skulle teste ut fornøyelsespark, men kollegaen var bare opptatt av bikinikroppen',
|
||||
'description': 'md5:1504a54606c4dde3e4e61fc97aa857e0',
|
||||
'description': 'md5:49cc8370e7d66e8a2ef15c3b4631fd3f',
|
||||
'thumbnail': r're:https?://.*\.jpg',
|
||||
'timestamp': 1404039863,
|
||||
'upload_date': '20140629',
|
||||
'duration': 69.544,
|
||||
'uploader_id': '1027729757001',
|
||||
'upload_date': '20160916',
|
||||
'duration': 69,
|
||||
'uploader_id': 'UCk5pvsyZJoYJBd7_oFPTlRQ',
|
||||
'uploader': 'Dagbladet',
|
||||
},
|
||||
'add_ie': ['BrightcoveNew']
|
||||
'add_ie': ['Youtube']
|
||||
}, {
|
||||
'url': 'http://dbtv.no/3649835190001',
|
||||
'url': 'https://www.dagbladet.no/video/embed/xlGmyIeN9Jo/?autoplay=false',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://www.dbtv.no/lazyplayer/4631135248001',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://dbtv.no/vice/5000634109001',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'http://dbtv.no/filmtrailer/3359293614001',
|
||||
'url': 'https://www.dagbladet.no/video/truer-iran-bor-passe-dere/PalfB2Cw',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
@staticmethod
|
||||
def _extract_urls(webpage):
|
||||
return [url for _, url in re.findall(
|
||||
r'<iframe[^>]+src=(["\'])((?:https?:)?//(?:www\.)?dbtv\.no/(?:lazy)?player/\d+.*?)\1',
|
||||
r'<iframe[^>]+src=(["\'])((?:https?:)?//(?:www\.)?dagbladet\.no/video/embed/(?:[0-9A-Za-z_-]{11}|[a-zA-Z0-9]{8}).*?)\1',
|
||||
webpage)]
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id, display_id = re.match(self._VALID_URL, url).groups()
|
||||
|
||||
return {
|
||||
display_id, video_id = re.match(self._VALID_URL, url).groups()
|
||||
info = {
|
||||
'_type': 'url_transparent',
|
||||
'url': 'http://players.brightcove.net/1027729757001/default_default/index.html?videoId=%s' % video_id,
|
||||
'id': video_id,
|
||||
'display_id': display_id,
|
||||
'ie_key': 'BrightcoveNew',
|
||||
}
|
||||
if len(video_id) == 11:
|
||||
info.update({
|
||||
'url': video_id,
|
||||
'ie_key': 'Youtube',
|
||||
})
|
||||
else:
|
||||
info.update({
|
||||
'url': 'jwplatform:' + video_id,
|
||||
'ie_key': 'JWPlatform',
|
||||
})
|
||||
return info
|
||||
|
||||
@ -19,9 +19,9 @@ from ..compat import compat_HTTPError
|
||||
class DiscoveryIE(DiscoveryGoBaseIE):
|
||||
_VALID_URL = r'''(?x)https?://
|
||||
(?P<site>
|
||||
(?:(?:www|go)\.)?discovery|
|
||||
(?:www\.)?
|
||||
(?:
|
||||
discovery|
|
||||
investigationdiscovery|
|
||||
discoverylife|
|
||||
animalplanet|
|
||||
@ -56,6 +56,9 @@ class DiscoveryIE(DiscoveryGoBaseIE):
|
||||
}, {
|
||||
'url': 'https://www.investigationdiscovery.com/tv-shows/final-vision/full-episodes/final-vision',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://go.discovery.com/tv-shows/alaskan-bush-people/videos/follow-your-own-road',
|
||||
'only_matching': True,
|
||||
}]
|
||||
_GEO_COUNTRIES = ['US']
|
||||
_GEO_BYPASS = False
|
||||
|
||||
94
youtube_dl/extractor/dlive.py
Normal file
94
youtube_dl/extractor/dlive.py
Normal file
@ -0,0 +1,94 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import int_or_none
|
||||
|
||||
|
||||
class DLiveVODIE(InfoExtractor):
|
||||
IE_NAME = 'dlive:vod'
|
||||
_VALID_URL = r'https?://(?:www\.)?dlive\.tv/p/(?P<uploader_id>.+?)\+(?P<id>[a-zA-Z0-9]+)'
|
||||
_TEST = {
|
||||
'url': 'https://dlive.tv/p/pdp+3mTzOl4WR',
|
||||
'info_dict': {
|
||||
'id': '3mTzOl4WR',
|
||||
'ext': 'mp4',
|
||||
'title': 'Minecraft with james charles epic',
|
||||
'upload_date': '20190701',
|
||||
'timestamp': 1562011015,
|
||||
'uploader_id': 'pdp',
|
||||
}
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
uploader_id, vod_id = re.match(self._VALID_URL, url).groups()
|
||||
broadcast = self._download_json(
|
||||
'https://graphigo.prd.dlive.tv/', vod_id,
|
||||
data=json.dumps({'query': '''query {
|
||||
pastBroadcast(permlink:"%s+%s") {
|
||||
content
|
||||
createdAt
|
||||
length
|
||||
playbackUrl
|
||||
title
|
||||
thumbnailUrl
|
||||
viewCount
|
||||
}
|
||||
}''' % (uploader_id, vod_id)}).encode())['data']['pastBroadcast']
|
||||
title = broadcast['title']
|
||||
formats = self._extract_m3u8_formats(
|
||||
broadcast['playbackUrl'], vod_id, 'mp4', 'm3u8_native')
|
||||
self._sort_formats(formats)
|
||||
return {
|
||||
'id': vod_id,
|
||||
'title': title,
|
||||
'uploader_id': uploader_id,
|
||||
'formats': formats,
|
||||
'description': broadcast.get('content'),
|
||||
'thumbnail': broadcast.get('thumbnailUrl'),
|
||||
'timestamp': int_or_none(broadcast.get('createdAt'), 1000),
|
||||
'view_count': int_or_none(broadcast.get('viewCount')),
|
||||
}
|
||||
|
||||
|
||||
class DLiveStreamIE(InfoExtractor):
|
||||
IE_NAME = 'dlive:stream'
|
||||
_VALID_URL = r'https?://(?:www\.)?dlive\.tv/(?!p/)(?P<id>[\w.-]+)'
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_name = self._match_id(url)
|
||||
user = self._download_json(
|
||||
'https://graphigo.prd.dlive.tv/', display_name,
|
||||
data=json.dumps({'query': '''query {
|
||||
userByDisplayName(displayname:"%s") {
|
||||
livestream {
|
||||
content
|
||||
createdAt
|
||||
title
|
||||
thumbnailUrl
|
||||
watchingCount
|
||||
}
|
||||
username
|
||||
}
|
||||
}''' % display_name}).encode())['data']['userByDisplayName']
|
||||
livestream = user['livestream']
|
||||
title = livestream['title']
|
||||
username = user['username']
|
||||
formats = self._extract_m3u8_formats(
|
||||
'https://live.prd.dlive.tv/hls/live/%s.m3u8' % username,
|
||||
display_name, 'mp4')
|
||||
self._sort_formats(formats)
|
||||
return {
|
||||
'id': display_name,
|
||||
'title': self._live_title(title),
|
||||
'uploader': display_name,
|
||||
'uploader_id': username,
|
||||
'formats': formats,
|
||||
'description': livestream.get('content'),
|
||||
'thumbnail': livestream.get('thumbnailUrl'),
|
||||
'is_live': True,
|
||||
'timestamp': int_or_none(livestream.get('createdAt'), 1000),
|
||||
'view_count': int_or_none(livestream.get('watchingCount')),
|
||||
}
|
||||
@ -2,6 +2,7 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
@ -18,7 +19,7 @@ from ..utils import (
|
||||
|
||||
|
||||
class EinthusanIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://einthusan\.tv/movie/watch/(?P<id>[^/?#&]+)'
|
||||
_VALID_URL = r'https?://(?P<host>einthusan\.(?:tv|com))/movie/watch/(?P<id>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'https://einthusan.tv/movie/watch/9097/',
|
||||
'md5': 'ff0f7f2065031b8a2cf13a933731c035',
|
||||
@ -32,6 +33,9 @@ class EinthusanIE(InfoExtractor):
|
||||
}, {
|
||||
'url': 'https://einthusan.tv/movie/watch/51MZ/?lang=hindi',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://einthusan.com/movie/watch/9097/',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
# reversed from jsoncrypto.prototype.decrypt() in einthusan-PGMovieWatcher.js
|
||||
@ -41,7 +45,9 @@ class EinthusanIE(InfoExtractor):
|
||||
)).decode('utf-8'), video_id)
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
host = mobj.group('host')
|
||||
video_id = mobj.group('id')
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
@ -53,7 +59,7 @@ class EinthusanIE(InfoExtractor):
|
||||
page_id = self._html_search_regex(
|
||||
'<html[^>]+data-pageid="([^"]+)"', webpage, 'page ID')
|
||||
video_data = self._download_json(
|
||||
'https://einthusan.tv/ajax/movie/watch/%s/' % video_id, video_id,
|
||||
'https://%s/ajax/movie/watch/%s/' % (host, video_id), video_id,
|
||||
data=urlencode_postdata({
|
||||
'xEvent': 'UIVideoPlayer.PingOutcome',
|
||||
'xJson': json.dumps({
|
||||
|
||||
@ -216,17 +216,14 @@ class FiveThirtyEightIE(InfoExtractor):
|
||||
_TEST = {
|
||||
'url': 'http://fivethirtyeight.com/features/how-the-6-8-raiders-can-still-make-the-playoffs/',
|
||||
'info_dict': {
|
||||
'id': '21846851',
|
||||
'ext': 'mp4',
|
||||
'id': '56032156',
|
||||
'ext': 'flv',
|
||||
'title': 'FiveThirtyEight: The Raiders can still make the playoffs',
|
||||
'description': 'Neil Paine breaks down the simplest scenario that will put the Raiders into the playoffs at 8-8.',
|
||||
'timestamp': 1513960621,
|
||||
'upload_date': '20171222',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
'expected_warnings': ['Unable to download f4m manifest'],
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
@ -234,9 +231,8 @@ class FiveThirtyEightIE(InfoExtractor):
|
||||
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
video_id = self._search_regex(
|
||||
r'data-video-id=["\'](?P<id>\d+)',
|
||||
webpage, 'video id', group='id')
|
||||
embed_url = self._search_regex(
|
||||
r'<iframe[^>]+src=["\'](https?://fivethirtyeight\.abcnews\.go\.com/video/embed/\d+/\d+)',
|
||||
webpage, 'embed url')
|
||||
|
||||
return self.url_result(
|
||||
'http://espn.go.com/video/clip?id=%s' % video_id, ESPNIE.ie_key())
|
||||
return self.url_result(embed_url, 'AbcNewsVideo')
|
||||
|
||||
@ -395,11 +395,7 @@ from .frontendmasters import (
|
||||
FrontendMastersCourseIE
|
||||
)
|
||||
from .funimation import FunimationIE
|
||||
from .funk import (
|
||||
FunkMixIE,
|
||||
FunkChannelIE,
|
||||
)
|
||||
from .funnyordie import FunnyOrDieIE
|
||||
from .funk import FunkIE
|
||||
from .fusion import FusionIE
|
||||
from .fxnetworks import FXNetworksIE
|
||||
from .gaia import GaiaIE
|
||||
@ -583,6 +579,7 @@ from .linkedin import (
|
||||
)
|
||||
from .linuxacademy import LinuxAcademyIE
|
||||
from .litv import LiTVIE
|
||||
from .livejournal import LiveJournalIE
|
||||
from .liveleak import (
|
||||
LiveLeakIE,
|
||||
LiveLeakEmbedIE,
|
||||
@ -971,7 +968,6 @@ from .rts import RTSIE
|
||||
from .rtve import RTVEALaCartaIE, RTVELiveIE, RTVEInfantilIE, RTVELiveIE, RTVETelevisionIE
|
||||
from .rtvnh import RTVNHIE
|
||||
from .rtvs import RTVSIE
|
||||
from .rudo import RudoIE
|
||||
from .ruhd import RUHDIE
|
||||
from .rutube import (
|
||||
RutubeIE,
|
||||
@ -1259,6 +1255,10 @@ from .udn import UDNEmbedIE
|
||||
from .ufctv import UFCTVIE
|
||||
from .uktvplay import UKTVPlayIE
|
||||
from .digiteka import DigitekaIE
|
||||
from .dlive import (
|
||||
DLiveVODIE,
|
||||
DLiveStreamIE,
|
||||
)
|
||||
from .umg import UMGDeIE
|
||||
from .unistra import UnistraIE
|
||||
from .unity import UnityIE
|
||||
@ -1448,6 +1448,7 @@ from .yahoo import (
|
||||
YahooSearchIE,
|
||||
YahooGyaOPlayerIE,
|
||||
YahooGyaOIE,
|
||||
YahooJapanNewsIE,
|
||||
)
|
||||
from .yandexdisk import YandexDiskIE
|
||||
from .yandexmusic import (
|
||||
|
||||
@ -428,7 +428,7 @@ class FacebookIE(InfoExtractor):
|
||||
timestamp = int_or_none(self._search_regex(
|
||||
r'<abbr[^>]+data-utime=["\'](\d+)', webpage,
|
||||
'timestamp', default=None))
|
||||
thumbnail = self._og_search_thumbnail(webpage)
|
||||
thumbnail = self._html_search_meta(['og:image', 'twitter:image'], webpage)
|
||||
|
||||
view_count = parse_count(self._search_regex(
|
||||
r'\bviewCount\s*:\s*["\']([\d,.]+)', webpage, 'view count',
|
||||
|
||||
@ -9,7 +9,7 @@ from ..utils import int_or_none
|
||||
|
||||
class FiveTVIE(InfoExtractor):
|
||||
_VALID_URL = r'''(?x)
|
||||
http://
|
||||
https?://
|
||||
(?:www\.)?5-tv\.ru/
|
||||
(?:
|
||||
(?:[^/]+/)+(?P<id>\d+)|
|
||||
@ -39,6 +39,7 @@ class FiveTVIE(InfoExtractor):
|
||||
'duration': 180,
|
||||
},
|
||||
}, {
|
||||
# redirect to https://www.5-tv.ru/projects/1000095/izvestia-glavnoe/
|
||||
'url': 'http://www.5-tv.ru/glavnoe/#itemDetails',
|
||||
'info_dict': {
|
||||
'id': 'glavnoe',
|
||||
@ -46,6 +47,7 @@ class FiveTVIE(InfoExtractor):
|
||||
'title': r're:^Итоги недели с \d+ по \d+ \w+ \d{4} года$',
|
||||
'thumbnail': r're:^https?://.*\.jpg$',
|
||||
},
|
||||
'skip': 'redirect to «Известия. Главное» project page',
|
||||
}, {
|
||||
'url': 'http://www.5-tv.ru/glavnoe/broadcasts/508645/',
|
||||
'only_matching': True,
|
||||
@ -70,7 +72,7 @@ class FiveTVIE(InfoExtractor):
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
video_url = self._search_regex(
|
||||
[r'<div[^>]+?class="flowplayer[^>]+?data-href="([^"]+)"',
|
||||
[r'<div[^>]+?class="(?:flow)?player[^>]+?data-href="([^"]+)"',
|
||||
r'<a[^>]+?href="([^"]+)"[^>]+?class="videoplayer"'],
|
||||
webpage, 'video url')
|
||||
|
||||
|
||||
@ -1,89 +1,21 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import itertools
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from .nexx import NexxIE
|
||||
from ..compat import compat_str
|
||||
from ..utils import (
|
||||
int_or_none,
|
||||
try_get,
|
||||
str_or_none,
|
||||
)
|
||||
|
||||
|
||||
class FunkBaseIE(InfoExtractor):
|
||||
_HEADERS = {
|
||||
'Accept': '*/*',
|
||||
'Accept-Language': 'en-US,en;q=0.9,ru;q=0.8',
|
||||
'authorization': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnROYW1lIjoid2ViYXBwLXYzMSIsInNjb3BlIjoic3RhdGljLWNvbnRlbnQtYXBpLGN1cmF0aW9uLWFwaSxuZXh4LWNvbnRlbnQtYXBpLXYzMSx3ZWJhcHAtYXBpIn0.mbuG9wS9Yf5q6PqgR4fiaRFIagiHk9JhwoKES7ksVX4',
|
||||
}
|
||||
_AUTH = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnROYW1lIjoid2ViYXBwLXYzMSIsInNjb3BlIjoic3RhdGljLWNvbnRlbnQtYXBpLGN1cmF0aW9uLWFwaSxuZXh4LWNvbnRlbnQtYXBpLXYzMSx3ZWJhcHAtYXBpIn0.mbuG9wS9Yf5q6PqgR4fiaRFIagiHk9JhwoKES7ksVX4'
|
||||
|
||||
@staticmethod
|
||||
def _make_headers(referer):
|
||||
headers = FunkBaseIE._HEADERS.copy()
|
||||
headers['Referer'] = referer
|
||||
return headers
|
||||
|
||||
def _make_url_result(self, video):
|
||||
return {
|
||||
'_type': 'url_transparent',
|
||||
'url': 'nexx:741:%s' % video['sourceId'],
|
||||
'ie_key': NexxIE.ie_key(),
|
||||
'id': video['sourceId'],
|
||||
'title': video.get('title'),
|
||||
'description': video.get('description'),
|
||||
'duration': int_or_none(video.get('duration')),
|
||||
'season_number': int_or_none(video.get('seasonNr')),
|
||||
'episode_number': int_or_none(video.get('episodeNr')),
|
||||
}
|
||||
|
||||
|
||||
class FunkMixIE(FunkBaseIE):
|
||||
_VALID_URL = r'https?://(?:www\.)?funk\.net/mix/(?P<id>[^/]+)/(?P<alias>[^/?#&]+)'
|
||||
class FunkIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?funk\.net/(?:channel|playlist)/[^/]+/(?P<display_id>[0-9a-z-]+)-(?P<id>\d+)'
|
||||
_TESTS = [{
|
||||
'url': 'https://www.funk.net/mix/59d65d935f8b160001828b5b/die-realste-kifferdoku-aller-zeiten',
|
||||
'md5': '8edf617c2f2b7c9847dfda313f199009',
|
||||
'info_dict': {
|
||||
'id': '123748',
|
||||
'ext': 'mp4',
|
||||
'title': '"Die realste Kifferdoku aller Zeiten"',
|
||||
'description': 'md5:c97160f5bafa8d47ec8e2e461012aa9d',
|
||||
'timestamp': 1490274721,
|
||||
'upload_date': '20170323',
|
||||
},
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
mix_id = mobj.group('id')
|
||||
alias = mobj.group('alias')
|
||||
|
||||
lists = self._download_json(
|
||||
'https://www.funk.net/api/v3.1/curation/curatedLists/',
|
||||
mix_id, headers=self._make_headers(url), query={
|
||||
'size': 100,
|
||||
})['_embedded']['curatedListList']
|
||||
|
||||
metas = next(
|
||||
l for l in lists
|
||||
if mix_id in (l.get('entityId'), l.get('alias')))['videoMetas']
|
||||
video = next(
|
||||
meta['videoDataDelegate']
|
||||
for meta in metas
|
||||
if try_get(
|
||||
meta, lambda x: x['videoDataDelegate']['alias'],
|
||||
compat_str) == alias)
|
||||
|
||||
return self._make_url_result(video)
|
||||
|
||||
|
||||
class FunkChannelIE(FunkBaseIE):
|
||||
_VALID_URL = r'https?://(?:www\.)?funk\.net/channel/(?P<id>[^/]+)/(?P<alias>[^/?#&]+)'
|
||||
_TESTS = [{
|
||||
'url': 'https://www.funk.net/channel/ba/die-lustigsten-instrumente-aus-dem-internet-teil-2',
|
||||
'url': 'https://www.funk.net/channel/ba-793/die-lustigsten-instrumente-aus-dem-internet-teil-2-1155821',
|
||||
'md5': '8dd9d9ab59b4aa4173b3197f2ea48e81',
|
||||
'info_dict': {
|
||||
'id': '1155821',
|
||||
'ext': 'mp4',
|
||||
@ -92,83 +24,26 @@ class FunkChannelIE(FunkBaseIE):
|
||||
'timestamp': 1514507395,
|
||||
'upload_date': '20171229',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
|
||||
}, {
|
||||
# only available via byIdList API
|
||||
'url': 'https://www.funk.net/channel/informr/martin-sonneborn-erklaert-die-eu',
|
||||
'info_dict': {
|
||||
'id': '205067',
|
||||
'ext': 'mp4',
|
||||
'title': 'Martin Sonneborn erklärt die EU',
|
||||
'description': 'md5:050f74626e4ed87edf4626d2024210c0',
|
||||
'timestamp': 1494424042,
|
||||
'upload_date': '20170510',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'https://www.funk.net/channel/59d5149841dca100012511e3/mein-erster-job-lovemilla-folge-1/lovemilla/',
|
||||
'url': 'https://www.funk.net/playlist/neuesteVideos/kameras-auf-dem-fusion-festival-1618699',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
channel_id = mobj.group('id')
|
||||
alias = mobj.group('alias')
|
||||
|
||||
headers = self._make_headers(url)
|
||||
|
||||
video = None
|
||||
|
||||
# Id-based channels are currently broken on their side: webplayer
|
||||
# tries to process them via byChannelAlias endpoint and fails
|
||||
# predictably.
|
||||
for page_num in itertools.count():
|
||||
by_channel_alias = self._download_json(
|
||||
'https://www.funk.net/api/v3.1/webapp/videos/byChannelAlias/%s'
|
||||
% channel_id,
|
||||
'Downloading byChannelAlias JSON page %d' % (page_num + 1),
|
||||
headers=headers, query={
|
||||
'filterFsk': 'false',
|
||||
'sort': 'creationDate,desc',
|
||||
'size': 100,
|
||||
'page': page_num,
|
||||
}, fatal=False)
|
||||
if not by_channel_alias:
|
||||
break
|
||||
video_list = try_get(
|
||||
by_channel_alias, lambda x: x['_embedded']['videoList'], list)
|
||||
if not video_list:
|
||||
break
|
||||
try:
|
||||
video = next(r for r in video_list if r.get('alias') == alias)
|
||||
break
|
||||
except StopIteration:
|
||||
pass
|
||||
if not try_get(
|
||||
by_channel_alias, lambda x: x['_links']['next']):
|
||||
break
|
||||
|
||||
if not video:
|
||||
by_id_list = self._download_json(
|
||||
'https://www.funk.net/api/v3.0/content/videos/byIdList',
|
||||
channel_id, 'Downloading byIdList JSON', headers=headers,
|
||||
query={
|
||||
'ids': alias,
|
||||
}, fatal=False)
|
||||
if by_id_list:
|
||||
video = try_get(by_id_list, lambda x: x['result'][0], dict)
|
||||
|
||||
if not video:
|
||||
results = self._download_json(
|
||||
'https://www.funk.net/api/v3.0/content/videos/filter',
|
||||
channel_id, 'Downloading filter JSON', headers=headers, query={
|
||||
'channelId': channel_id,
|
||||
'size': 100,
|
||||
})['result']
|
||||
video = next(r for r in results if r.get('alias') == alias)
|
||||
|
||||
return self._make_url_result(video)
|
||||
display_id, nexx_id = re.match(self._VALID_URL, url).groups()
|
||||
video = self._download_json(
|
||||
'https://www.funk.net/api/v4.0/videos/' + nexx_id, nexx_id)
|
||||
return {
|
||||
'_type': 'url_transparent',
|
||||
'url': 'nexx:741:' + nexx_id,
|
||||
'ie_key': NexxIE.ie_key(),
|
||||
'id': nexx_id,
|
||||
'title': video.get('title'),
|
||||
'description': video.get('description'),
|
||||
'duration': int_or_none(video.get('duration')),
|
||||
'channel_id': str_or_none(video.get('channelId')),
|
||||
'display_id': display_id,
|
||||
'tags': video.get('tags'),
|
||||
'thumbnail': video.get('imageUrlLandscape'),
|
||||
}
|
||||
|
||||
@ -1,162 +0,0 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
float_or_none,
|
||||
int_or_none,
|
||||
unified_timestamp,
|
||||
)
|
||||
|
||||
|
||||
class FunnyOrDieIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?funnyordie\.com/(?P<type>embed|articles|videos)/(?P<id>[0-9a-f]+)(?:$|[?#/])'
|
||||
_TESTS = [{
|
||||
'url': 'http://www.funnyordie.com/videos/0732f586d7/heart-shaped-box-literal-video-version',
|
||||
'md5': 'bcd81e0c4f26189ee09be362ad6e6ba9',
|
||||
'info_dict': {
|
||||
'id': '0732f586d7',
|
||||
'ext': 'mp4',
|
||||
'title': 'Heart-Shaped Box: Literal Video Version',
|
||||
'description': 'md5:ea09a01bc9a1c46d9ab696c01747c338',
|
||||
'thumbnail': r're:^http:.*\.jpg$',
|
||||
'uploader': 'DASjr',
|
||||
'timestamp': 1317904928,
|
||||
'upload_date': '20111006',
|
||||
'duration': 318.3,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://www.funnyordie.com/embed/e402820827',
|
||||
'info_dict': {
|
||||
'id': 'e402820827',
|
||||
'ext': 'mp4',
|
||||
'title': 'Please Use This Song (Jon Lajoie)',
|
||||
'description': 'Please use this to sell something. www.jonlajoie.com',
|
||||
'thumbnail': r're:^http:.*\.jpg$',
|
||||
'timestamp': 1398988800,
|
||||
'upload_date': '20140502',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://www.funnyordie.com/articles/ebf5e34fc8/10-hours-of-walking-in-nyc-as-a-man',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
|
||||
video_id = mobj.group('id')
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
|
||||
links = re.findall(r'<source src="([^"]+/v)[^"]+\.([^"]+)" type=\'video', webpage)
|
||||
if not links:
|
||||
raise ExtractorError('No media links available for %s' % video_id)
|
||||
|
||||
links.sort(key=lambda link: 1 if link[1] == 'mp4' else 0)
|
||||
|
||||
m3u8_url = self._search_regex(
|
||||
r'<source[^>]+src=(["\'])(?P<url>.+?/master\.m3u8[^"\']*)\1',
|
||||
webpage, 'm3u8 url', group='url')
|
||||
|
||||
formats = []
|
||||
|
||||
m3u8_formats = self._extract_m3u8_formats(
|
||||
m3u8_url, video_id, 'mp4', 'm3u8_native',
|
||||
m3u8_id='hls', fatal=False)
|
||||
source_formats = list(filter(
|
||||
lambda f: f.get('vcodec') != 'none', m3u8_formats))
|
||||
|
||||
bitrates = [int(bitrate) for bitrate in re.findall(r'[,/]v(\d+)(?=[,/])', m3u8_url)]
|
||||
bitrates.sort()
|
||||
|
||||
if source_formats:
|
||||
self._sort_formats(source_formats)
|
||||
|
||||
for bitrate, f in zip(bitrates, source_formats or [{}] * len(bitrates)):
|
||||
for path, ext in links:
|
||||
ff = f.copy()
|
||||
if ff:
|
||||
if ext != 'mp4':
|
||||
ff = dict(
|
||||
[(k, v) for k, v in ff.items()
|
||||
if k in ('height', 'width', 'format_id')])
|
||||
ff.update({
|
||||
'format_id': ff['format_id'].replace('hls', ext),
|
||||
'ext': ext,
|
||||
'protocol': 'http',
|
||||
})
|
||||
else:
|
||||
ff.update({
|
||||
'format_id': '%s-%d' % (ext, bitrate),
|
||||
'vbr': bitrate,
|
||||
})
|
||||
ff['url'] = self._proto_relative_url(
|
||||
'%s%d.%s' % (path, bitrate, ext))
|
||||
formats.append(ff)
|
||||
self._check_formats(formats, video_id)
|
||||
|
||||
formats.extend(m3u8_formats)
|
||||
self._sort_formats(
|
||||
formats, field_preference=('height', 'width', 'tbr', 'format_id'))
|
||||
|
||||
subtitles = {}
|
||||
for src, src_lang in re.findall(r'<track kind="captions" src="([^"]+)" srclang="([^"]+)"', webpage):
|
||||
subtitles[src_lang] = [{
|
||||
'ext': src.split('/')[-1],
|
||||
'url': 'http://www.funnyordie.com%s' % src,
|
||||
}]
|
||||
|
||||
timestamp = unified_timestamp(self._html_search_meta(
|
||||
'uploadDate', webpage, 'timestamp', default=None))
|
||||
|
||||
uploader = self._html_search_regex(
|
||||
r'<h\d[^>]+\bclass=["\']channel-preview-name[^>]+>(.+?)</h',
|
||||
webpage, 'uploader', default=None)
|
||||
|
||||
title, description, thumbnail, duration = [None] * 4
|
||||
|
||||
medium = self._parse_json(
|
||||
self._search_regex(
|
||||
r'jsonMedium\s*=\s*({.+?});', webpage, 'JSON medium',
|
||||
default='{}'),
|
||||
video_id, fatal=False)
|
||||
if medium:
|
||||
title = medium.get('title')
|
||||
duration = float_or_none(medium.get('duration'))
|
||||
if not timestamp:
|
||||
timestamp = unified_timestamp(medium.get('publishDate'))
|
||||
|
||||
post = self._parse_json(
|
||||
self._search_regex(
|
||||
r'fb_post\s*=\s*(\{.*?\});', webpage, 'post details',
|
||||
default='{}'),
|
||||
video_id, fatal=False)
|
||||
if post:
|
||||
if not title:
|
||||
title = post.get('name')
|
||||
description = post.get('description')
|
||||
thumbnail = post.get('picture')
|
||||
|
||||
if not title:
|
||||
title = self._og_search_title(webpage)
|
||||
if not description:
|
||||
description = self._og_search_description(webpage)
|
||||
if not duration:
|
||||
duration = int_or_none(self._html_search_meta(
|
||||
('video:duration', 'duration'), webpage, 'duration', default=False))
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'uploader': uploader,
|
||||
'timestamp': timestamp,
|
||||
'duration': duration,
|
||||
'formats': formats,
|
||||
'subtitles': subtitles,
|
||||
}
|
||||
@ -1,12 +1,19 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .brightcove import BrightcoveNewIE
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
clean_html,
|
||||
get_element_by_class,
|
||||
get_element_by_id,
|
||||
)
|
||||
|
||||
|
||||
class GameInformerIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?gameinformer\.com/(?:[^/]+/)*(?P<id>.+)\.aspx'
|
||||
_TEST = {
|
||||
_VALID_URL = r'https?://(?:www\.)?gameinformer\.com/(?:[^/]+/)*(?P<id>[^.?&#]+)'
|
||||
_TESTS = [{
|
||||
# normal Brightcove embed code extracted with BrightcoveNewIE._extract_url
|
||||
'url': 'http://www.gameinformer.com/b/features/archive/2015/09/26/replay-animal-crossing.aspx',
|
||||
'md5': '292f26da1ab4beb4c9099f1304d2b071',
|
||||
'info_dict': {
|
||||
@ -18,16 +25,25 @@ class GameInformerIE(InfoExtractor):
|
||||
'upload_date': '20150928',
|
||||
'uploader_id': '694940074001',
|
||||
},
|
||||
}
|
||||
}, {
|
||||
# Brightcove id inside unique element with field--name-field-brightcove-video-id class
|
||||
'url': 'https://www.gameinformer.com/video-feature/new-gameplay-today/2019/07/09/new-gameplay-today-streets-of-rogue',
|
||||
'info_dict': {
|
||||
'id': '6057111913001',
|
||||
'ext': 'mp4',
|
||||
'title': 'New Gameplay Today – Streets Of Rogue',
|
||||
'timestamp': 1562699001,
|
||||
'upload_date': '20190709',
|
||||
'uploader_id': '694940074001',
|
||||
|
||||
},
|
||||
}]
|
||||
BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/694940074001/default_default/index.html?videoId=%s'
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
webpage = self._download_webpage(
|
||||
url, display_id, headers=self.geo_verification_headers())
|
||||
brightcove_id = self._search_regex(
|
||||
[r'<[^>]+\bid=["\']bc_(\d+)', r"getVideo\('[^']+video_id=(\d+)"],
|
||||
webpage, 'brightcove id')
|
||||
return self.url_result(
|
||||
self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id, 'BrightcoveNew',
|
||||
brightcove_id)
|
||||
brightcove_id = clean_html(get_element_by_class('field--name-field-brightcove-video-id', webpage) or get_element_by_id('video-source-content', webpage))
|
||||
brightcove_url = self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id if brightcove_id else BrightcoveNewIE._extract_url(self, webpage)
|
||||
return self.url_result(brightcove_url, 'BrightcoveNew', brightcove_id)
|
||||
|
||||
@ -2226,7 +2226,7 @@ class GenericIE(InfoExtractor):
|
||||
default_search = 'fixup_error'
|
||||
|
||||
if default_search in ('auto', 'auto_warning', 'fixup_error'):
|
||||
if '/' in url:
|
||||
if re.match(r'^[^\s/]+\.[^\s/]+/', url):
|
||||
self._downloader.report_warning('The url doesn\'t specify the protocol, trying with http')
|
||||
return self.url_result('http://' + url)
|
||||
elif default_search != 'fixup_error':
|
||||
|
||||
@ -11,7 +11,7 @@ from ..utils import (
|
||||
|
||||
|
||||
class GfycatIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?gfycat\.com/(?:ifr/|gifs/detail/)?(?P<id>[^-/?#]+)'
|
||||
_VALID_URL = r'https?://(?:www\.)?gfycat\.com/(?:ru/|ifr/|gifs/detail/)?(?P<id>[^-/?#]+)'
|
||||
_TESTS = [{
|
||||
'url': 'http://gfycat.com/DeadlyDecisiveGermanpinscher',
|
||||
'info_dict': {
|
||||
@ -44,6 +44,9 @@ class GfycatIE(InfoExtractor):
|
||||
'categories': list,
|
||||
'age_limit': 0,
|
||||
}
|
||||
}, {
|
||||
'url': 'https://gfycat.com/ru/RemarkableDrearyAmurstarfish',
|
||||
'only_matching': True
|
||||
}, {
|
||||
'url': 'https://gfycat.com/gifs/detail/UnconsciousLankyIvorygull',
|
||||
'only_matching': True
|
||||
|
||||
@ -103,6 +103,11 @@ class KalturaIE(InfoExtractor):
|
||||
{
|
||||
'url': 'https://www.kaltura.com:443/index.php/extwidget/preview/partner_id/1770401/uiconf_id/37307382/entry_id/0_58u8kme7/embed/iframe?&flashvars[streamerType]=auto',
|
||||
'only_matching': True,
|
||||
},
|
||||
{
|
||||
# unavailable source format
|
||||
'url': 'kaltura:513551:1_66x4rg7o',
|
||||
'only_matching': True,
|
||||
}
|
||||
]
|
||||
|
||||
@ -306,12 +311,17 @@ class KalturaIE(InfoExtractor):
|
||||
f['fileExt'] = 'mp4'
|
||||
video_url = sign_url(
|
||||
'%s/flavorId/%s' % (data_url, f['id']))
|
||||
format_id = '%(fileExt)s-%(bitrate)s' % f
|
||||
# Source format may not be available (e.g. kaltura:513551:1_66x4rg7o)
|
||||
if f.get('isOriginal') is True and not self._is_valid_url(
|
||||
video_url, entry_id, format_id):
|
||||
continue
|
||||
# audio-only has no videoCodecId (e.g. kaltura:1926081:0_c03e1b5g
|
||||
# -f mp4-56)
|
||||
vcodec = 'none' if 'videoCodecId' not in f and f.get(
|
||||
'frameRate') == 0 else f.get('videoCodecId')
|
||||
formats.append({
|
||||
'format_id': '%(fileExt)s-%(bitrate)s' % f,
|
||||
'format_id': format_id,
|
||||
'ext': f.get('fileExt'),
|
||||
'tbr': int_or_none(f['bitrate']),
|
||||
'fps': int_or_none(f.get('frameRate')),
|
||||
|
||||
@ -326,7 +326,7 @@ class LetvCloudIE(InfoExtractor):
|
||||
elif play_json.get('code'):
|
||||
raise ExtractorError('Letv cloud returned error %d' % play_json['code'], expected=True)
|
||||
else:
|
||||
raise ExtractorError('Letv cloud returned an unknwon error')
|
||||
raise ExtractorError('Letv cloud returned an unknown error')
|
||||
|
||||
def b64decode(s):
|
||||
return compat_b64decode(s).decode('utf-8')
|
||||
|
||||
42
youtube_dl/extractor/livejournal.py
Normal file
42
youtube_dl/extractor/livejournal.py
Normal file
@ -0,0 +1,42 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import compat_str
|
||||
from ..utils import int_or_none
|
||||
|
||||
|
||||
class LiveJournalIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:[^.]+\.)?livejournal\.com/video/album/\d+.+?\bid=(?P<id>\d+)'
|
||||
_TEST = {
|
||||
'url': 'https://andrei-bt.livejournal.com/video/album/407/?mode=view&id=51272',
|
||||
'md5': 'adaf018388572ced8a6f301ace49d4b2',
|
||||
'info_dict': {
|
||||
'id': '1263729',
|
||||
'ext': 'mp4',
|
||||
'title': 'Истребители против БПЛА',
|
||||
'upload_date': '20190624',
|
||||
'timestamp': 1561406715,
|
||||
}
|
||||
}
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
record = self._parse_json(self._search_regex(
|
||||
r'Site\.page\s*=\s*({.+?});', webpage,
|
||||
'page data'), video_id)['video']['record']
|
||||
storage_id = compat_str(record['storageid'])
|
||||
title = record.get('name')
|
||||
if title:
|
||||
# remove filename extension(.mp4, .mov, etc...)
|
||||
title = title.rsplit('.', 1)[0]
|
||||
return {
|
||||
'_type': 'url_transparent',
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'thumbnail': record.get('thumbnail'),
|
||||
'timestamp': int_or_none(record.get('timecreate')),
|
||||
'url': 'eagleplatform:vc.videos.livejournal.com:' + storage_id,
|
||||
'ie_key': 'EaglePlatform',
|
||||
}
|
||||
@ -117,6 +117,10 @@ class LyndaIE(LyndaBaseIE):
|
||||
}, {
|
||||
'url': 'https://www.lynda.com/de/Graphic-Design-tutorials/Willkommen-Grundlagen-guten-Gestaltung/393570/393572-4.html',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
# Status="NotFound", Message="Transcript not found"
|
||||
'url': 'https://www.lynda.com/ASP-NET-tutorials/What-you-should-know/5034180/2811512-4.html',
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
def _raise_unavailable(self, video_id):
|
||||
@ -247,12 +251,17 @@ class LyndaIE(LyndaBaseIE):
|
||||
|
||||
def _get_subtitles(self, video_id):
|
||||
url = 'https://www.lynda.com/ajax/player?videoId=%s&type=transcript' % video_id
|
||||
subs = self._download_json(url, None, False)
|
||||
subs = self._download_webpage(
|
||||
url, video_id, 'Downloading subtitles JSON', fatal=False)
|
||||
if not subs or 'Status="NotFound"' in subs:
|
||||
return {}
|
||||
subs = self._parse_json(subs, video_id, fatal=False)
|
||||
if not subs:
|
||||
return {}
|
||||
fixed_subs = self._fix_subtitles(subs)
|
||||
if fixed_subs:
|
||||
return {'en': [{'ext': 'srt', 'data': fixed_subs}]}
|
||||
else:
|
||||
return {}
|
||||
return {}
|
||||
|
||||
|
||||
class LyndaCourseIE(LyndaBaseIE):
|
||||
|
||||
@ -79,6 +79,9 @@ class MGTVIE(InfoExtractor):
|
||||
'ext': 'mp4',
|
||||
'tbr': tbr,
|
||||
'protocol': 'm3u8_native',
|
||||
'http_headers': {
|
||||
'Referer': url,
|
||||
},
|
||||
})
|
||||
self._sort_formats(formats)
|
||||
|
||||
|
||||
@ -39,7 +39,12 @@ class Porn91IE(InfoExtractor):
|
||||
r'<div id="viewvideo-title">([^<]+)</div>', webpage, 'title')
|
||||
title = title.replace('\n', '')
|
||||
|
||||
info_dict = self._parse_html5_media_entries(url, webpage, video_id)[0]
|
||||
video_link_url = self._search_regex(
|
||||
r'<textarea[^>]+id=["\']fm-video_link[^>]+>([^<]+)</textarea>',
|
||||
webpage, 'video link')
|
||||
videopage = self._download_webpage(video_link_url, video_id)
|
||||
|
||||
info_dict = self._parse_html5_media_entries(url, videopage, video_id)[0]
|
||||
|
||||
duration = parse_duration(self._search_regex(
|
||||
r'时长:\s*</span>\s*(\d+:\d+)', webpage, 'duration', fatal=False))
|
||||
|
||||
@ -4,11 +4,14 @@ from __future__ import unicode_literals
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..compat import (
|
||||
compat_HTTPError,
|
||||
compat_str,
|
||||
)
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
strip_or_none,
|
||||
unescapeHTML,
|
||||
str_or_none,
|
||||
urlencode_postdata,
|
||||
)
|
||||
|
||||
@ -21,15 +24,14 @@ class RoosterTeethIE(InfoExtractor):
|
||||
'url': 'http://roosterteeth.com/episode/million-dollars-but-season-2-million-dollars-but-the-game-announcement',
|
||||
'md5': 'e2bd7764732d785ef797700a2489f212',
|
||||
'info_dict': {
|
||||
'id': '26576',
|
||||
'id': '9156',
|
||||
'display_id': 'million-dollars-but-season-2-million-dollars-but-the-game-announcement',
|
||||
'ext': 'mp4',
|
||||
'title': 'Million Dollars, But...: Million Dollars, But... The Game Announcement',
|
||||
'description': 'md5:0cc3b21986d54ed815f5faeccd9a9ca5',
|
||||
'title': 'Million Dollars, But... The Game Announcement',
|
||||
'description': 'md5:168a54b40e228e79f4ddb141e89fe4f5',
|
||||
'thumbnail': r're:^https?://.*\.png$',
|
||||
'series': 'Million Dollars, But...',
|
||||
'episode': 'Million Dollars, But... The Game Announcement',
|
||||
'comment_count': int,
|
||||
},
|
||||
}, {
|
||||
'url': 'http://achievementhunter.roosterteeth.com/episode/off-topic-the-achievement-hunter-podcast-2016-i-didn-t-think-it-would-pass-31',
|
||||
@ -89,60 +91,55 @@ class RoosterTeethIE(InfoExtractor):
|
||||
|
||||
def _real_extract(self, url):
|
||||
display_id = self._match_id(url)
|
||||
api_episode_url = 'https://svod-be.roosterteeth.com/api/v1/episodes/%s' % display_id
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
episode = strip_or_none(unescapeHTML(self._search_regex(
|
||||
(r'videoTitle\s*=\s*(["\'])(?P<title>(?:(?!\1).)+)\1',
|
||||
r'<title>(?P<title>[^<]+)</title>'), webpage, 'title',
|
||||
default=None, group='title')))
|
||||
|
||||
title = strip_or_none(self._og_search_title(
|
||||
webpage, default=None)) or episode
|
||||
|
||||
m3u8_url = self._search_regex(
|
||||
r'file\s*:\s*(["\'])(?P<url>http.+?\.m3u8.*?)\1',
|
||||
webpage, 'm3u8 url', default=None, group='url')
|
||||
|
||||
if not m3u8_url:
|
||||
if re.search(r'<div[^>]+class=["\']non-sponsor', webpage):
|
||||
self.raise_login_required(
|
||||
'%s is only available for FIRST members' % display_id)
|
||||
|
||||
if re.search(r'<div[^>]+class=["\']golive-gate', webpage):
|
||||
self.raise_login_required('%s is not available yet' % display_id)
|
||||
|
||||
raise ExtractorError('Unable to extract m3u8 URL')
|
||||
try:
|
||||
m3u8_url = self._download_json(
|
||||
api_episode_url + '/videos', display_id,
|
||||
'Downloading video JSON metadata')['data'][0]['attributes']['url']
|
||||
except ExtractorError as e:
|
||||
if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
|
||||
if self._parse_json(e.cause.read().decode(), display_id).get('access') is False:
|
||||
self.raise_login_required(
|
||||
'%s is only available for FIRST members' % display_id)
|
||||
raise
|
||||
|
||||
formats = self._extract_m3u8_formats(
|
||||
m3u8_url, display_id, ext='mp4',
|
||||
entry_protocol='m3u8_native', m3u8_id='hls')
|
||||
m3u8_url, display_id, 'mp4', 'm3u8_native', m3u8_id='hls')
|
||||
self._sort_formats(formats)
|
||||
|
||||
description = strip_or_none(self._og_search_description(webpage))
|
||||
thumbnail = self._proto_relative_url(self._og_search_thumbnail(webpage))
|
||||
episode = self._download_json(
|
||||
api_episode_url, display_id,
|
||||
'Downloading episode JSON metadata')['data'][0]
|
||||
attributes = episode['attributes']
|
||||
title = attributes.get('title') or attributes['display_title']
|
||||
video_id = compat_str(episode['id'])
|
||||
|
||||
series = self._search_regex(
|
||||
(r'<h2>More ([^<]+)</h2>', r'<a[^>]+>See All ([^<]+) Videos<'),
|
||||
webpage, 'series', fatal=False)
|
||||
|
||||
comment_count = int_or_none(self._search_regex(
|
||||
r'>Comments \((\d+)\)<', webpage,
|
||||
'comment count', fatal=False))
|
||||
|
||||
video_id = self._search_regex(
|
||||
(r'containerId\s*=\s*["\']episode-(\d+)\1',
|
||||
r'<div[^<]+id=["\']episode-(\d+)'), webpage,
|
||||
'video id', default=display_id)
|
||||
thumbnails = []
|
||||
for image in episode.get('included', {}).get('images', []):
|
||||
if image.get('type') == 'episode_image':
|
||||
img_attributes = image.get('attributes') or {}
|
||||
for k in ('thumb', 'small', 'medium', 'large'):
|
||||
img_url = img_attributes.get(k)
|
||||
if img_url:
|
||||
thumbnails.append({
|
||||
'id': k,
|
||||
'url': img_url,
|
||||
})
|
||||
|
||||
return {
|
||||
'id': video_id,
|
||||
'display_id': display_id,
|
||||
'title': title,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'series': series,
|
||||
'episode': episode,
|
||||
'comment_count': comment_count,
|
||||
'description': attributes.get('description') or attributes.get('caption'),
|
||||
'thumbnails': thumbnails,
|
||||
'series': attributes.get('show_title'),
|
||||
'season_number': int_or_none(attributes.get('season_number')),
|
||||
'season_id': attributes.get('season_id'),
|
||||
'episode': title,
|
||||
'episode_number': int_or_none(attributes.get('number')),
|
||||
'episode_id': str_or_none(episode.get('uuid')),
|
||||
'formats': formats,
|
||||
'channel_id': attributes.get('channel_id'),
|
||||
'duration': int_or_none(attributes.get('length')),
|
||||
}
|
||||
|
||||
@ -32,7 +32,7 @@ class RtlNlIE(InfoExtractor):
|
||||
'duration': 1167.96,
|
||||
},
|
||||
}, {
|
||||
# best format avaialble a3t
|
||||
# best format available a3t
|
||||
'url': 'http://www.rtl.nl/system/videoplayer/derden/rtlnieuws/video_embed.html#uuid=84ae5571-ac25-4225-ae0c-ef8d9efb2aed/autoplay=false',
|
||||
'md5': 'dea7474214af1271d91ef332fb8be7ea',
|
||||
'info_dict': {
|
||||
|
||||
@ -1,53 +0,0 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
js_to_json,
|
||||
get_element_by_class,
|
||||
unified_strdate,
|
||||
)
|
||||
|
||||
|
||||
class RudoIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://rudo\.video/vod/(?P<id>[0-9a-zA-Z]+)'
|
||||
|
||||
_TEST = {
|
||||
'url': 'http://rudo.video/vod/oTzw0MGnyG',
|
||||
'md5': '2a03a5b32dd90a04c83b6d391cf7b415',
|
||||
'info_dict': {
|
||||
'id': 'oTzw0MGnyG',
|
||||
'ext': 'mp4',
|
||||
'title': 'Comentario Tomás Mosciatti',
|
||||
'upload_date': '20160617',
|
||||
},
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _extract_url(cls, webpage):
|
||||
mobj = re.search(
|
||||
r'<iframe[^>]+src=(?P<q1>[\'"])(?P<url>(?:https?:)?//rudo\.video/vod/[0-9a-zA-Z]+)(?P=q1)',
|
||||
webpage)
|
||||
if mobj:
|
||||
return mobj.group('url')
|
||||
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
|
||||
webpage = self._download_webpage(url, video_id, encoding='iso-8859-1')
|
||||
|
||||
jwplayer_data = self._parse_json(self._search_regex(
|
||||
r'(?s)playerInstance\.setup\(({.+?})\)', webpage, 'jwplayer data'), video_id,
|
||||
transform_source=lambda s: js_to_json(re.sub(r'encodeURI\([^)]+\)', '""', s)))
|
||||
|
||||
info_dict = self._parse_jwplayer_data(
|
||||
jwplayer_data, video_id, require_title=False, m3u8_id='hls', mpd_id='dash')
|
||||
|
||||
info_dict.update({
|
||||
'title': self._og_search_title(webpage),
|
||||
'upload_date': unified_strdate(get_element_by_class('date', webpage)),
|
||||
})
|
||||
|
||||
return info_dict
|
||||
@ -197,7 +197,7 @@ class SoundcloudIE(InfoExtractor):
|
||||
'skip_download': True,
|
||||
},
|
||||
},
|
||||
# not avaialble via api.soundcloud.com/i1/tracks/id/streams
|
||||
# not available via api.soundcloud.com/i1/tracks/id/streams
|
||||
{
|
||||
'url': 'https://soundcloud.com/giovannisarani/mezzo-valzer',
|
||||
'md5': 'e22aecd2bc88e0e4e432d7dcc0a1abf7',
|
||||
|
||||
@ -5,6 +5,7 @@ import re
|
||||
from .common import InfoExtractor
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
merge_dicts,
|
||||
orderedSet,
|
||||
parse_duration,
|
||||
parse_resolution,
|
||||
@ -26,6 +27,8 @@ class SpankBangIE(InfoExtractor):
|
||||
'description': 'dillion harper masturbates on a bed',
|
||||
'thumbnail': r're:^https?://.*\.jpg$',
|
||||
'uploader': 'silly2587',
|
||||
'timestamp': 1422571989,
|
||||
'upload_date': '20150129',
|
||||
'age_limit': 18,
|
||||
}
|
||||
}, {
|
||||
@ -106,31 +109,36 @@ class SpankBangIE(InfoExtractor):
|
||||
|
||||
for format_id, format_url in stream.items():
|
||||
if format_id.startswith(STREAM_URL_PREFIX):
|
||||
if format_url and isinstance(format_url, list):
|
||||
format_url = format_url[0]
|
||||
extract_format(
|
||||
format_id[len(STREAM_URL_PREFIX):], format_url)
|
||||
|
||||
self._sort_formats(formats)
|
||||
|
||||
info = self._search_json_ld(webpage, video_id, default={})
|
||||
|
||||
title = self._html_search_regex(
|
||||
r'(?s)<h1[^>]*>(.+?)</h1>', webpage, 'title')
|
||||
r'(?s)<h1[^>]*>(.+?)</h1>', webpage, 'title', default=None)
|
||||
description = self._search_regex(
|
||||
r'<div[^>]+\bclass=["\']bottom[^>]+>\s*<p>[^<]*</p>\s*<p>([^<]+)',
|
||||
webpage, 'description', fatal=False)
|
||||
thumbnail = self._og_search_thumbnail(webpage)
|
||||
uploader = self._search_regex(
|
||||
r'class="user"[^>]*><img[^>]+>([^<]+)',
|
||||
webpage, 'description', default=None)
|
||||
thumbnail = self._og_search_thumbnail(webpage, default=None)
|
||||
uploader = self._html_search_regex(
|
||||
(r'(?s)<li[^>]+class=["\']profile[^>]+>(.+?)</a>',
|
||||
r'class="user"[^>]*><img[^>]+>([^<]+)'),
|
||||
webpage, 'uploader', default=None)
|
||||
duration = parse_duration(self._search_regex(
|
||||
r'<div[^>]+\bclass=["\']right_side[^>]+>\s*<span>([^<]+)',
|
||||
webpage, 'duration', fatal=False))
|
||||
webpage, 'duration', default=None))
|
||||
view_count = str_to_int(self._search_regex(
|
||||
r'([\d,.]+)\s+plays', webpage, 'view count', fatal=False))
|
||||
r'([\d,.]+)\s+plays', webpage, 'view count', default=None))
|
||||
|
||||
age_limit = self._rta_search(webpage)
|
||||
|
||||
return {
|
||||
return merge_dicts({
|
||||
'id': video_id,
|
||||
'title': title,
|
||||
'title': title or video_id,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'uploader': uploader,
|
||||
@ -138,7 +146,8 @@ class SpankBangIE(InfoExtractor):
|
||||
'view_count': view_count,
|
||||
'formats': formats,
|
||||
'age_limit': age_limit,
|
||||
}
|
||||
}, info
|
||||
)
|
||||
|
||||
|
||||
class SpankBangPlaylistIE(InfoExtractor):
|
||||
|
||||
@ -22,7 +22,7 @@ class BellatorIE(MTVServicesInfoExtractor):
|
||||
'only_matching': True,
|
||||
}]
|
||||
|
||||
_FEED_URL = 'http://www.spike.com/feeds/mrss/'
|
||||
_FEED_URL = 'http://www.bellator.com/feeds/mrss/'
|
||||
_GEO_COUNTRIES = ['US']
|
||||
|
||||
|
||||
|
||||
@ -428,11 +428,22 @@ class TwitterIE(InfoExtractor):
|
||||
'params': {
|
||||
'skip_download': True, # requires ffmpeg
|
||||
},
|
||||
}, {
|
||||
'url': 'https://twitter.com/foobar/status/1087791357756956680',
|
||||
'info_dict': {
|
||||
'id': '1087791357756956680',
|
||||
'ext': 'mp4',
|
||||
'title': 'Twitter - A new is coming. Some of you got an opt-in to try it now. Check out the emoji button, quick keyboard shortcuts, upgraded trends, advanced search, and more. Let us know your thoughts!',
|
||||
'thumbnail': r're:^https?://.*\.jpg',
|
||||
'description': 'md5:66d493500c013e3e2d434195746a7f78',
|
||||
'uploader': 'Twitter',
|
||||
'uploader_id': 'Twitter',
|
||||
'duration': 61.567,
|
||||
},
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
user_id = mobj.group('user_id')
|
||||
twid = mobj.group('id')
|
||||
|
||||
webpage, urlh = self._download_webpage_handle(
|
||||
@ -441,8 +452,13 @@ class TwitterIE(InfoExtractor):
|
||||
if 'twitter.com/account/suspended' in urlh.geturl():
|
||||
raise ExtractorError('Account suspended by Twitter.', expected=True)
|
||||
|
||||
if user_id is None:
|
||||
mobj = re.match(self._VALID_URL, urlh.geturl())
|
||||
user_id = None
|
||||
|
||||
redirect_mobj = re.match(self._VALID_URL, urlh.geturl())
|
||||
if redirect_mobj:
|
||||
user_id = redirect_mobj.group('user_id')
|
||||
|
||||
if not user_id:
|
||||
user_id = mobj.group('user_id')
|
||||
|
||||
username = remove_end(self._og_search_title(webpage), ' on Twitter')
|
||||
|
||||
@ -4,7 +4,10 @@ from __future__ import unicode_literals
|
||||
from .common import InfoExtractor
|
||||
from .once import OnceIE
|
||||
from ..compat import compat_urllib_parse_unquote
|
||||
from ..utils import ExtractorError
|
||||
from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
)
|
||||
|
||||
|
||||
class VoxMediaVolumeIE(OnceIE):
|
||||
@ -13,18 +16,43 @@ class VoxMediaVolumeIE(OnceIE):
|
||||
def _real_extract(self, url):
|
||||
video_id = self._match_id(url)
|
||||
webpage = self._download_webpage(url, video_id)
|
||||
video_data = self._parse_json(self._search_regex(
|
||||
r'Volume\.createVideo\(({.+})\s*,\s*{.*}\s*,\s*\[.*\]\s*,\s*{.*}\);', webpage, 'video data'), video_id)
|
||||
|
||||
setup = self._parse_json(self._search_regex(
|
||||
r'setup\s*=\s*({.+});', webpage, 'setup'), video_id)
|
||||
video_data = setup.get('video') or {}
|
||||
info = {
|
||||
'id': video_id,
|
||||
'title': video_data.get('title_short'),
|
||||
'description': video_data.get('description_long') or video_data.get('description_short'),
|
||||
'thumbnail': video_data.get('brightcove_thumbnail')
|
||||
}
|
||||
asset = setup.get('asset') or setup.get('params') or {}
|
||||
|
||||
formats = []
|
||||
hls_url = asset.get('hls_url')
|
||||
if hls_url:
|
||||
formats.extend(self._extract_m3u8_formats(
|
||||
hls_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
|
||||
mp4_url = asset.get('mp4_url')
|
||||
if mp4_url:
|
||||
tbr = self._search_regex(r'-(\d+)k\.', mp4_url, 'bitrate', default=None)
|
||||
format_id = 'http'
|
||||
if tbr:
|
||||
format_id += '-' + tbr
|
||||
formats.append({
|
||||
'format_id': format_id,
|
||||
'url': mp4_url,
|
||||
'tbr': int_or_none(tbr),
|
||||
})
|
||||
if formats:
|
||||
self._sort_formats(formats)
|
||||
info['formats'] = formats
|
||||
return info
|
||||
|
||||
for provider_video_type in ('ooyala', 'youtube', 'brightcove'):
|
||||
provider_video_id = video_data.get('%s_id' % provider_video_type)
|
||||
if not provider_video_id:
|
||||
continue
|
||||
info = {
|
||||
'id': video_id,
|
||||
'title': video_data.get('title_short'),
|
||||
'description': video_data.get('description_long') or video_data.get('description_short'),
|
||||
'thumbnail': video_data.get('brightcove_thumbnail')
|
||||
}
|
||||
if provider_video_type == 'brightcove':
|
||||
info['formats'] = self._extract_once_formats(provider_video_id)
|
||||
self._sort_formats(info['formats'])
|
||||
@ -39,46 +67,49 @@ class VoxMediaVolumeIE(OnceIE):
|
||||
|
||||
|
||||
class VoxMediaIE(InfoExtractor):
|
||||
_VALID_URL = r'https?://(?:www\.)?(?:(?:theverge|vox|sbnation|eater|polygon|curbed|racked)\.com|recode\.net)/(?:[^/]+/)*(?P<id>[^/?]+)'
|
||||
_VALID_URL = r'https?://(?:www\.)?(?:(?:theverge|vox|sbnation|eater|polygon|curbed|racked|funnyordie)\.com|recode\.net)/(?:[^/]+/)*(?P<id>[^/?]+)'
|
||||
_TESTS = [{
|
||||
# Volume embed, Youtube
|
||||
'url': 'http://www.theverge.com/2014/6/27/5849272/material-world-how-google-discovered-what-software-is-made-of',
|
||||
'info_dict': {
|
||||
'id': '11eXZobjrG8DCSTgrNjVinU-YmmdYjhe',
|
||||
'id': 'j4mLW6x17VM',
|
||||
'ext': 'mp4',
|
||||
'title': 'Google\'s new material design direction',
|
||||
'description': 'md5:2f44f74c4d14a1f800ea73e1c6832ad2',
|
||||
'title': 'Material world: how Google discovered what software is made of',
|
||||
'description': 'md5:dfc17e7715e3b542d66e33a109861382',
|
||||
'upload_date': '20190710',
|
||||
'uploader_id': 'TheVerge',
|
||||
'uploader': 'The Verge',
|
||||
},
|
||||
'params': {
|
||||
# m3u8 download
|
||||
'skip_download': True,
|
||||
},
|
||||
'add_ie': ['Ooyala'],
|
||||
'add_ie': ['Youtube'],
|
||||
}, {
|
||||
# data-ooyala-id
|
||||
# Volume embed, Youtube
|
||||
'url': 'http://www.theverge.com/2014/10/21/7025853/google-nexus-6-hands-on-photos-video-android-phablet',
|
||||
'md5': 'd744484ff127884cd2ba09e3fa604e4b',
|
||||
'md5': '4c8f4a0937752b437c3ebc0ed24802b5',
|
||||
'info_dict': {
|
||||
'id': 'RkZXU4cTphOCPDMZg5oEounJyoFI0g-B',
|
||||
'id': 'Gy8Md3Eky38',
|
||||
'ext': 'mp4',
|
||||
'title': 'The Nexus 6: hands-on with Google\'s phablet',
|
||||
'description': 'md5:87a51fe95ff8cea8b5bdb9ac7ae6a6af',
|
||||
'description': 'md5:d9f0216e5fb932dd2033d6db37ac3f1d',
|
||||
'uploader_id': 'TheVerge',
|
||||
'upload_date': '20141021',
|
||||
'uploader': 'The Verge',
|
||||
},
|
||||
'add_ie': ['Ooyala'],
|
||||
'skip': 'Video Not Found',
|
||||
'add_ie': ['Youtube'],
|
||||
'skip': 'similar to the previous test',
|
||||
}, {
|
||||
# volume embed
|
||||
# Volume embed, Youtube
|
||||
'url': 'http://www.vox.com/2016/3/31/11336640/mississippi-lgbt-religious-freedom-bill',
|
||||
'info_dict': {
|
||||
'id': 'wydzk3dDpmRz7PQoXRsTIX6XTkPjYL0b',
|
||||
'id': 'YCjDnX-Xzhg',
|
||||
'ext': 'mp4',
|
||||
'title': 'The new frontier of LGBTQ civil rights, explained',
|
||||
'description': 'md5:0dc58e94a465cbe91d02950f770eb93f',
|
||||
'title': "Mississippi's laws are so bad that its anti-LGBTQ law isn't needed to allow discrimination",
|
||||
'description': 'md5:fc1317922057de31cd74bce91eb1c66c',
|
||||
'uploader_id': 'voxdotcom',
|
||||
'upload_date': '20150915',
|
||||
'uploader': 'Vox',
|
||||
},
|
||||
'params': {
|
||||
# m3u8 download
|
||||
'skip_download': True,
|
||||
},
|
||||
'add_ie': ['Ooyala'],
|
||||
'add_ie': ['Youtube'],
|
||||
'skip': 'similar to the previous test',
|
||||
}, {
|
||||
# youtube embed
|
||||
'url': 'http://www.vox.com/2016/3/24/11291692/robot-dance',
|
||||
@ -93,6 +124,7 @@ class VoxMediaIE(InfoExtractor):
|
||||
'uploader': 'Vox',
|
||||
},
|
||||
'add_ie': ['Youtube'],
|
||||
'skip': 'Page no longer contain videos',
|
||||
}, {
|
||||
# SBN.VideoLinkset.entryGroup multiple ooyala embeds
|
||||
'url': 'http://www.sbnation.com/college-football-recruiting/2015/2/3/7970291/national-signing-day-rationalizations-itll-be-ok-itll-be-ok',
|
||||
@ -118,10 +150,11 @@ class VoxMediaIE(InfoExtractor):
|
||||
'description': 'md5:e02d56b026d51aa32c010676765a690d',
|
||||
},
|
||||
}],
|
||||
'skip': 'Page no longer contain videos',
|
||||
}, {
|
||||
# volume embed, Brightcove Once
|
||||
'url': 'https://www.recode.net/2014/6/17/11628066/post-post-pc-ceo-the-full-code-conference-video-of-microsofts-satya',
|
||||
'md5': '01571a896281f77dc06e084138987ea2',
|
||||
'md5': '2dbc77b8b0bff1894c2fce16eded637d',
|
||||
'info_dict': {
|
||||
'id': '1231c973d',
|
||||
'ext': 'mp4',
|
||||
|
||||
@ -64,7 +64,15 @@ class VRVBaseIE(InfoExtractor):
|
||||
|
||||
def _call_cms(self, path, video_id, note):
|
||||
if not self._CMS_SIGNING:
|
||||
self._CMS_SIGNING = self._call_api('index', video_id, 'CMS Signing')['cms_signing']
|
||||
index = self._call_api('index', video_id, 'CMS Signing')
|
||||
self._CMS_SIGNING = index.get('cms_signing') or {}
|
||||
if not self._CMS_SIGNING:
|
||||
for signing_policy in index.get('signing_policies', []):
|
||||
signing_path = signing_policy.get('path')
|
||||
if signing_path and signing_path.startswith('/cms/'):
|
||||
name, value = signing_policy.get('name'), signing_policy.get('value')
|
||||
if name and value:
|
||||
self._CMS_SIGNING[name] = value
|
||||
return self._download_json(
|
||||
self._API_DOMAIN + path, video_id, query=self._CMS_SIGNING,
|
||||
note='Downloading %s JSON metadata' % note, headers=self.geo_verification_headers())
|
||||
|
||||
@ -1,12 +1,14 @@
|
||||
# coding: utf-8
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
import re
|
||||
|
||||
from .common import InfoExtractor, SearchInfoExtractor
|
||||
from ..compat import (
|
||||
compat_str,
|
||||
compat_urllib_parse,
|
||||
compat_urlparse,
|
||||
)
|
||||
@ -18,7 +20,9 @@ from ..utils import (
|
||||
int_or_none,
|
||||
mimetype2ext,
|
||||
smuggle_url,
|
||||
try_get,
|
||||
unescapeHTML,
|
||||
url_or_none,
|
||||
)
|
||||
|
||||
from .brightcove import (
|
||||
@ -556,3 +560,130 @@ class YahooGyaOIE(InfoExtractor):
|
||||
'https://gyao.yahoo.co.jp/player/%s/' % video_id.replace(':', '/'),
|
||||
YahooGyaOPlayerIE.ie_key(), video_id))
|
||||
return self.playlist_result(entries, program_id)
|
||||
|
||||
|
||||
class YahooJapanNewsIE(InfoExtractor):
|
||||
IE_NAME = 'yahoo:japannews'
|
||||
IE_DESC = 'Yahoo! Japan News'
|
||||
_VALID_URL = r'https?://(?P<host>(?:news|headlines)\.yahoo\.co\.jp)[^\d]*(?P<id>\d[\d-]*\d)?'
|
||||
_GEO_COUNTRIES = ['JP']
|
||||
_TESTS = [{
|
||||
'url': 'https://headlines.yahoo.co.jp/videonews/ann?a=20190716-00000071-ann-int',
|
||||
'info_dict': {
|
||||
'id': '1736242',
|
||||
'ext': 'mp4',
|
||||
'title': 'ムン大統領が対日批判を強化“現金化”効果は？（テレビ朝日系（ANN）） - Yahoo!ニュース',
|
||||
'description': '韓国の元徴用工らを巡る裁判の原告が弁護士が差し押さえた三菱重工業の資産を売却して - Yahoo!ニュース(テレビ朝日系（ANN）)',
|
||||
'thumbnail': r're:^https?://.*\.[a-zA-Z\d]{3,4}$',
|
||||
},
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
# geo restricted
|
||||
'url': 'https://headlines.yahoo.co.jp/hl?a=20190721-00000001-oxv-l04',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://headlines.yahoo.co.jp/videonews/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://news.yahoo.co.jp',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://news.yahoo.co.jp/byline/hashimotojunji/20190628-00131977/',
|
||||
'only_matching': True,
|
||||
}, {
|
||||
'url': 'https://news.yahoo.co.jp/feature/1356',
|
||||
'only_matching': True
|
||||
}]
|
||||
|
||||
def _extract_formats(self, json_data, content_id):
|
||||
formats = []
|
||||
|
||||
video_data = try_get(
|
||||
json_data,
|
||||
lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'],
|
||||
list)
|
||||
for vid in video_data or []:
|
||||
delivery = vid.get('delivery')
|
||||
url = url_or_none(vid.get('Url'))
|
||||
if not delivery or not url:
|
||||
continue
|
||||
elif delivery == 'hls':
|
||||
formats.extend(
|
||||
self._extract_m3u8_formats(
|
||||
url, content_id, 'mp4', 'm3u8_native',
|
||||
m3u8_id='hls', fatal=False))
|
||||
else:
|
||||
formats.append({
|
||||
'url': url,
|
||||
'format_id': 'http-%s' % compat_str(vid.get('bitrate', '')),
|
||||
'height': int_or_none(vid.get('height')),
|
||||
'width': int_or_none(vid.get('width')),
|
||||
'tbr': int_or_none(vid.get('bitrate')),
|
||||
})
|
||||
self._remove_duplicate_formats(formats)
|
||||
self._sort_formats(formats)
|
||||
|
||||
return formats
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
host = mobj.group('host')
|
||||
display_id = mobj.group('id') or host
|
||||
|
||||
webpage = self._download_webpage(url, display_id)
|
||||
|
||||
title = self._html_search_meta(
|
||||
['og:title', 'twitter:title'], webpage, 'title', default=None
|
||||
) or self._html_search_regex('<title>([^<]+)</title>', webpage, 'title')
|
||||
|
||||
if display_id == host:
|
||||
# Headline page (w/ multiple BC playlists) ('news.yahoo.co.jp', 'headlines.yahoo.co.jp/videonews/', ...)
|
||||
stream_plists = re.findall(r'plist=(\d+)', webpage) or re.findall(r'plist["\']:\s*["\']([^"\']+)', webpage)
|
||||
entries = [
|
||||
self.url_result(
|
||||
smuggle_url(
|
||||
'http://players.brightcove.net/5690807595001/HyZNerRl7_default/index.html?playlistId=%s' % plist_id,
|
||||
{'geo_countries': ['JP']}),
|
||||
ie='BrightcoveNew', video_id=plist_id)
|
||||
for plist_id in stream_plists]
|
||||
return self.playlist_result(entries, playlist_title=title)
|
||||
|
||||
# Article page
|
||||
description = self._html_search_meta(
|
||||
['og:description', 'description', 'twitter:description'],
|
||||
webpage, 'description', default=None)
|
||||
thumbnail = self._og_search_thumbnail(
|
||||
webpage, default=None) or self._html_search_meta(
|
||||
'twitter:image', webpage, 'thumbnail', default=None)
|
||||
space_id = self._search_regex([
|
||||
r'<script[^>]+class=["\']yvpub-player["\'][^>]+spaceid=([^&"\']+)',
|
||||
r'YAHOO\.JP\.srch\.\w+link\.onLoad[^;]+spaceID["\' ]*:["\' ]+([^"\']+)',
|
||||
r'<!--\s+SpaceID=(\d+)'
|
||||
], webpage, 'spaceid')
|
||||
|
||||
content_id = self._search_regex(
|
||||
r'<script[^>]+class=["\']yvpub-player["\'][^>]+contentid=(?P<contentid>[^&"\']+)',
|
||||
webpage, 'contentid', group='contentid')
|
||||
|
||||
json_data = self._download_json(
|
||||
'https://feapi-yvpub.yahooapis.jp/v1/content/%s' % content_id,
|
||||
content_id,
|
||||
query={
|
||||
'appid': 'dj0zaiZpPVZMTVFJR0FwZWpiMyZzPWNvbnN1bWVyc2VjcmV0Jng9YjU-',
|
||||
'output': 'json',
|
||||
'space_id': space_id,
|
||||
'domain': host,
|
||||
'ak': hashlib.md5('_'.join((space_id, host)).encode()).hexdigest(),
|
||||
'device_type': '1100',
|
||||
})
|
||||
formats = self._extract_formats(json_data, content_id)
|
||||
|
||||
return {
|
||||
'id': content_id,
|
||||
'title': title,
|
||||
'description': description,
|
||||
'thumbnail': thumbnail,
|
||||
'formats': formats,
|
||||
}
|
||||
|
||||
@ -10,6 +10,7 @@ from ..utils import (
|
||||
ExtractorError,
|
||||
int_or_none,
|
||||
float_or_none,
|
||||
try_get,
|
||||
)
|
||||
|
||||
|
||||
@ -51,23 +52,43 @@ class YandexMusicTrackIE(YandexMusicBaseIE):
|
||||
IE_DESC = 'Яндекс.Музыка - Трек'
|
||||
_VALID_URL = r'https?://music\.yandex\.(?:ru|kz|ua|by)/album/(?P<album_id>\d+)/track/(?P<id>\d+)'
|
||||
|
||||
_TEST = {
|
||||
_TESTS = [{
|
||||
'url': 'http://music.yandex.ru/album/540508/track/4878838',
|
||||
'md5': 'f496818aa2f60b6c0062980d2e00dc20',
|
||||
'info_dict': {
|
||||
'id': '4878838',
|
||||
'ext': 'mp3',
|
||||
'title': 'Carlo Ambrosio, Carlo Ambrosio & Fabio Di Bari - Gypsy Eyes 1',
|
||||
'title': 'Carlo Ambrosio & Fabio Di Bari - Gypsy Eyes 1',
|
||||
'filesize': 4628061,
|
||||
'duration': 193.04,
|
||||
'track': 'Gypsy Eyes 1',
|
||||
'album': 'Gypsy Soul',
|
||||
'album_artist': 'Carlo Ambrosio',
|
||||
'artist': 'Carlo Ambrosio, Carlo Ambrosio & Fabio Di Bari',
|
||||
'artist': 'Carlo Ambrosio & Fabio Di Bari',
|
||||
'release_year': 2009,
|
||||
},
|
||||
'skip': 'Travis CI servers blocked by YandexMusic',
|
||||
}
|
||||
}, {
|
||||
# multiple disks
|
||||
'url': 'http://music.yandex.ru/album/3840501/track/705105',
|
||||
'md5': 'ebe7b4e2ac7ac03fe11c19727ca6153e',
|
||||
'info_dict': {
|
||||
'id': '705105',
|
||||
'ext': 'mp3',
|
||||
'title': 'Hooverphonic - Sometimes',
|
||||
'filesize': 5743386,
|
||||
'duration': 239.27,
|
||||
'track': 'Sometimes',
|
||||
'album': 'The Best of Hooverphonic',
|
||||
'album_artist': 'Hooverphonic',
|
||||
'artist': 'Hooverphonic',
|
||||
'release_year': 2016,
|
||||
'genre': 'pop',
|
||||
'disc_number': 2,
|
||||
'track_number': 9,
|
||||
},
|
||||
'skip': 'Travis CI servers blocked by YandexMusic',
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
mobj = re.match(self._VALID_URL, url)
|
||||
@ -110,9 +131,21 @@ class YandexMusicTrackIE(YandexMusicBaseIE):
|
||||
'abr': int_or_none(download_data.get('bitrate')),
|
||||
}
|
||||
|
||||
def extract_artist_name(artist):
|
||||
decomposed = artist.get('decomposed')
|
||||
if not isinstance(decomposed, list):
|
||||
return artist['name']
|
||||
parts = [artist['name']]
|
||||
for element in decomposed:
|
||||
if isinstance(element, dict) and element.get('name'):
|
||||
parts.append(element['name'])
|
||||
elif isinstance(element, compat_str):
|
||||
parts.append(element)
|
||||
return ''.join(parts)
|
||||
|
||||
def extract_artist(artist_list):
|
||||
if artist_list and isinstance(artist_list, list):
|
||||
artists_names = [a['name'] for a in artist_list if a.get('name')]
|
||||
artists_names = [extract_artist_name(a) for a in artist_list if a.get('name')]
|
||||
if artists_names:
|
||||
return ', '.join(artists_names)
|
||||
|
||||
@ -121,10 +154,17 @@ class YandexMusicTrackIE(YandexMusicBaseIE):
|
||||
album = albums[0]
|
||||
if isinstance(album, dict):
|
||||
year = album.get('year')
|
||||
disc_number = int_or_none(try_get(
|
||||
album, lambda x: x['trackPosition']['volume']))
|
||||
track_number = int_or_none(try_get(
|
||||
album, lambda x: x['trackPosition']['index']))
|
||||
track_info.update({
|
||||
'album': album.get('title'),
|
||||
'album_artist': extract_artist(album.get('artists')),
|
||||
'release_year': int_or_none(year),
|
||||
'genre': album.get('genre'),
|
||||
'disc_number': disc_number,
|
||||
'track_number': track_number,
|
||||
})
|
||||
|
||||
track_artist = extract_artist(track.get('artists'))
|
||||
@ -152,7 +192,7 @@ class YandexMusicAlbumIE(YandexMusicPlaylistBaseIE):
|
||||
IE_DESC = 'Яндекс.Музыка - Альбом'
|
||||
_VALID_URL = r'https?://music\.yandex\.(?:ru|kz|ua|by)/album/(?P<id>\d+)/?(\?|$)'
|
||||
|
||||
_TEST = {
|
||||
_TESTS = [{
|
||||
'url': 'http://music.yandex.ru/album/540508',
|
||||
'info_dict': {
|
||||
'id': '540508',
|
||||
@ -160,7 +200,15 @@ class YandexMusicAlbumIE(YandexMusicPlaylistBaseIE):
|
||||
},
|
||||
'playlist_count': 50,
|
||||
'skip': 'Travis CI servers blocked by YandexMusic',
|
||||
}
|
||||
}, {
|
||||
'url': 'https://music.yandex.ru/album/3840501',
|
||||
'info_dict': {
|
||||
'id': '3840501',
|
||||
'title': 'Hooverphonic - The Best of Hooverphonic (2016)',
|
||||
},
|
||||
'playlist_count': 33,
|
||||
'skip': 'Travis CI servers blocked by YandexMusic',
|
||||
}]
|
||||
|
||||
def _real_extract(self, url):
|
||||
album_id = self._match_id(url)
|
||||
@ -169,7 +217,7 @@ class YandexMusicAlbumIE(YandexMusicPlaylistBaseIE):
|
||||
'http://music.yandex.ru/handlers/album.jsx?album=%s' % album_id,
|
||||
album_id, 'Downloading album JSON')
|
||||
|
||||
entries = self._build_playlist(album['volumes'][0])
|
||||
entries = self._build_playlist([track for volume in album['volumes'] for track in volume])
|
||||
|
||||
title = '%s - %s' % (album['artists'][0]['name'], album['title'])
|
||||
year = album.get('year')
|
||||
|
||||
@ -27,6 +27,7 @@ from ..compat import (
|
||||
compat_str,
|
||||
)
|
||||
from ..utils import (
|
||||
bool_or_none,
|
||||
clean_html,
|
||||
dict_get,
|
||||
error_to_compat_str,
|
||||
@ -116,6 +117,8 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
|
||||
'f.req': json.dumps(f_req),
|
||||
'flowName': 'GlifWebSignIn',
|
||||
'flowEntry': 'ServiceLogin',
|
||||
# TODO: reverse actual botguard identifier generation algo
|
||||
'bgRequest': '["identifier",""]',
|
||||
})
|
||||
return self._download_json(
|
||||
url, None, note=note, errnote=errnote,
|
||||
@ -368,10 +371,15 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
|
||||
(?:www\.)?hooktube\.com/|
|
||||
(?:www\.)?yourepeat\.com/|
|
||||
tube\.majestyc\.net/|
|
||||
# Invidious instances taken from https://github.com/omarroth/invidious/wiki/Invidious-Instances
|
||||
(?:(?:www|dev)\.)?invidio\.us/|
|
||||
(?:www\.)?invidiou\.sh/|
|
||||
(?:www\.)?invidious\.snopyta\.org/|
|
||||
(?:(?:www|no)\.)?invidiou\.sh/|
|
||||
(?:(?:www|fi|de)\.)?invidious\.snopyta\.org/|
|
||||
(?:www\.)?invidious\.kabi\.tk/|
|
||||
(?:www\.)?invidious\.enkirton\.net/|
|
||||
(?:www\.)?invidious\.13ad\.de/|
|
||||
(?:www\.)?invidious\.mastodon\.host/|
|
||||
(?:www\.)?tube\.poal\.co/|
|
||||
(?:www\.)?vid\.wxzm\.sx/|
|
||||
youtube\.googleapis\.com/) # the various hostnames, with wildcard subdomains
|
||||
(?:.*?\#/)? # handle anchor (#/) redirect urls
|
||||
@ -1812,16 +1820,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
|
||||
video_details = try_get(
|
||||
player_response, lambda x: x['videoDetails'], dict) or {}
|
||||
|
||||
# title
|
||||
if 'title' in video_info:
|
||||
video_title = video_info['title'][0]
|
||||
elif 'title' in player_response:
|
||||
video_title = video_details['title']
|
||||
else:
|
||||
video_title = video_info.get('title', [None])[0] or video_details.get('title')
|
||||
if not video_title:
|
||||
self._downloader.report_warning('Unable to extract video title')
|
||||
video_title = '_'
|
||||
|
||||
# description
|
||||
description_original = video_description = get_element_by_id("eow-description", video_webpage)
|
||||
if video_description:
|
||||
|
||||
@ -1846,11 +1849,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
|
||||
''', replace_url, video_description)
|
||||
video_description = clean_html(video_description)
|
||||
else:
|
||||
fd_mobj = re.search(r'<meta name="description" content="([^"]+)"', video_webpage)
|
||||
if fd_mobj:
|
||||
video_description = unescapeHTML(fd_mobj.group(1))
|
||||
else:
|
||||
video_description = ''
|
||||
video_description = self._html_search_meta('description', video_webpage) or video_details.get('shortDescription')
|
||||
|
||||
if not smuggled_data.get('force_singlefeed', False):
|
||||
if not self._downloader.params.get('noplaylist'):
|
||||
@ -1888,6 +1887,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
|
||||
if view_count is None and video_details:
|
||||
view_count = int_or_none(video_details.get('viewCount'))
|
||||
|
||||
if is_live is None:
|
||||
is_live = bool_or_none(video_details.get('isLive'))
|
||||
|
||||
# Check for "rental" videos
|
||||
if 'ypc_video_rental_bar_text' in video_info and 'author' not in video_info:
|
||||
raise ExtractorError('"rental" videos not supported. See https://github.com/ytdl-org/youtube-dl/issues/359 for more information.', expected=True)
|
||||
@ -2421,7 +2423,7 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
(%(playlist_id)s)
|
||||
)""" % {'playlist_id': YoutubeBaseInfoExtractor._PLAYLIST_ID_RE}
|
||||
_TEMPLATE_URL = 'https://www.youtube.com/playlist?list=%s'
|
||||
_VIDEO_RE = r'href="\s*/watch\?v=(?P<id>[0-9A-Za-z_-]{11})&[^"]*?index=(?P<index>\d+)(?:[^>]+>(?P<title>[^<]+))?'
|
||||
_VIDEO_RE = r'href="\s*/watch\?v=(?P<id>[0-9A-Za-z_-]{11})(?:&(?:[^"]*?index=(?P<index>\d+))?(?:[^>]+>(?P<title>[^<]+))?)?'
|
||||
IE_NAME = 'youtube:playlist'
|
||||
_TESTS = [{
|
||||
'url': 'https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re',
|
||||
@ -2444,6 +2446,8 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'title': '29C3: Not my department',
|
||||
'id': 'PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC',
|
||||
'uploader': 'Christiaan008',
|
||||
'uploader_id': 'ChRiStIaAn008',
|
||||
},
|
||||
'playlist_count': 95,
|
||||
}, {
|
||||
@ -2452,6 +2456,8 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'title': '[OLD]Team Fortress 2 (Class-based LP)',
|
||||
'id': 'PLBB231211A4F62143',
|
||||
'uploader': 'Wickydoo',
|
||||
'uploader_id': 'Wickydoo',
|
||||
},
|
||||
'playlist_mincount': 26,
|
||||
}, {
|
||||
@ -2460,6 +2466,8 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'title': 'Uploads from Cauchemar',
|
||||
'id': 'UUBABnxM4Ar9ten8Mdjj1j0Q',
|
||||
'uploader': 'Cauchemar',
|
||||
'uploader_id': 'Cauchemar89',
|
||||
},
|
||||
'playlist_mincount': 799,
|
||||
}, {
|
||||
@ -2477,13 +2485,17 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'title': 'JODA15',
|
||||
'id': 'PL6IaIsEjSbf96XFRuNccS_RuEXwNdsoEu',
|
||||
'uploader': 'milan',
|
||||
'uploader_id': 'UCEI1-PVPcYXjB73Hfelbmaw',
|
||||
}
|
||||
}, {
|
||||
'url': 'http://www.youtube.com/embed/_xDOZElKyNU?list=PLsyOSbh5bs16vubvKePAQ1x3PhKavfBIl',
|
||||
'playlist_mincount': 485,
|
||||
'info_dict': {
|
||||
'title': '2017 華語最新單曲 (2/24更新)',
|
||||
'title': '2018 Chinese New Singles (11/6 updated)',
|
||||
'id': 'PLsyOSbh5bs16vubvKePAQ1x3PhKavfBIl',
|
||||
'uploader': 'LBK',
|
||||
'uploader_id': 'sdragonfang',
|
||||
}
|
||||
}, {
|
||||
'note': 'Embedded SWF player',
|
||||
@ -2492,13 +2504,16 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'title': 'JODA7',
|
||||
'id': 'YN5VISEtHet5D4NEvfTd0zcgFk84NqFZ',
|
||||
}
|
||||
},
|
||||
'skip': 'This playlist does not exist',
|
||||
}, {
|
||||
'note': 'Buggy playlist: the webpage has a "Load more" button but it doesn\'t have more videos',
|
||||
'url': 'https://www.youtube.com/playlist?list=UUXw-G3eDE9trcvY2sBMM_aA',
|
||||
'info_dict': {
|
||||
'title': 'Uploads from Interstellar Movie',
|
||||
'id': 'UUXw-G3eDE9trcvY2sBMM_aA',
|
||||
'uploader': 'Interstellar Movie',
|
||||
'uploader_id': 'InterstellarMovie1',
|
||||
},
|
||||
'playlist_mincount': 21,
|
||||
}, {
|
||||
@ -2523,6 +2538,7 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'params': {
|
||||
'skip_download': True,
|
||||
},
|
||||
'skip': 'This video is not available.',
|
||||
'add_ie': [YoutubeIE.ie_key()],
|
||||
}, {
|
||||
'url': 'https://youtu.be/yeWKywCrFtk?list=PL2qgrgXsNUG5ig9cat4ohreBjYLAPC0J5',
|
||||
@ -2534,7 +2550,6 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'uploader_id': 'backuspagemuseum',
|
||||
'uploader_url': r're:https?://(?:www\.)?youtube\.com/user/backuspagemuseum',
|
||||
'upload_date': '20161008',
|
||||
'license': 'Standard YouTube License',
|
||||
'description': 'md5:800c0c78d5eb128500bffd4f0b4f2e8a',
|
||||
'categories': ['Nonprofits & Activism'],
|
||||
'tags': list,
|
||||
@ -2545,6 +2560,16 @@ class YoutubePlaylistIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'noplaylist': True,
|
||||
'skip_download': True,
|
||||
},
|
||||
}, {
|
||||
# https://github.com/ytdl-org/youtube-dl/issues/21844
|
||||
'url': 'https://www.youtube.com/playlist?list=PLzH6n4zXuckpfMu_4Ff8E7Z1behQks5ba',
|
||||
'info_dict': {
|
||||
'title': 'Data Analysis with Dr Mike Pound',
|
||||
'id': 'PLzH6n4zXuckpfMu_4Ff8E7Z1behQks5ba',
|
||||
'uploader_id': 'Computerphile',
|
||||
'uploader': 'Computerphile',
|
||||
},
|
||||
'playlist_mincount': 11,
|
||||
}, {
|
||||
'url': 'https://youtu.be/uWyaPkt-VOI?list=PL9D9FC436B881BA21',
|
||||
'only_matching': True,
|
||||
@ -2711,6 +2736,8 @@ class YoutubeChannelIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'id': 'UUKfVa3S1e4PHvxWcwyMMg8w',
|
||||
'title': 'Uploads from lex will',
|
||||
'uploader': 'lex will',
|
||||
'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
|
||||
}
|
||||
}, {
|
||||
'note': 'Age restricted channel',
|
||||
@ -2720,6 +2747,8 @@ class YoutubeChannelIE(YoutubePlaylistBaseInfoExtractor):
|
||||
'info_dict': {
|
||||
'id': 'UUs0ifCMCm1icqRbqhUINa0w',
|
||||
'title': 'Uploads from Deus Ex',
|
||||
'uploader': 'Deus Ex',
|
||||
'uploader_id': 'DeusExOfficial',
|
||||
},
|
||||
}, {
|
||||
'url': 'https://invidio.us/channel/UC23qupoDRn9YOAVzeoxjOQA',
|
||||
@ -2804,6 +2833,8 @@ class YoutubeUserIE(YoutubeChannelIE):
|
||||
'info_dict': {
|
||||
'id': 'UUfX55Sx5hEFjoC3cNs6mCUQ',
|
||||
'title': 'Uploads from The Linux Foundation',
|
||||
'uploader': 'The Linux Foundation',
|
||||
'uploader_id': 'TheLinuxFoundation',
|
||||
}
|
||||
}, {
|
||||
# Only available via https://www.youtube.com/c/12minuteathlete/videos
|
||||
@ -2813,6 +2844,8 @@ class YoutubeUserIE(YoutubeChannelIE):
|
||||
'info_dict': {
|
||||
'id': 'UUVjM-zV6_opMDx7WYxnjZiQ',
|
||||
'title': 'Uploads from 12 Minute Athlete',
|
||||
'uploader': '12 Minute Athlete',
|
||||
'uploader_id': 'the12minuteathlete',
|
||||
}
|
||||
}, {
|
||||
'url': 'ytuser:phihag',
|
||||
@ -2906,7 +2939,7 @@ class YoutubePlaylistsIE(YoutubePlaylistsBaseInfoExtractor):
|
||||
'playlist_mincount': 4,
|
||||
'info_dict': {
|
||||
'id': 'ThirstForScience',
|
||||
'title': 'Thirst for Science',
|
||||
'title': 'ThirstForScience',
|
||||
},
|
||||
}, {
|
||||
# with "Load more" button
|
||||
@ -2923,6 +2956,7 @@ class YoutubePlaylistsIE(YoutubePlaylistsBaseInfoExtractor):
|
||||
'id': 'UCiU1dHvZObB2iP6xkJ__Icw',
|
||||
'title': 'Chem Player',
|
||||
},
|
||||
'skip': 'Blocked',
|
||||
}]
|
||||
|
||||
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
from __future__ import unicode_literals
|
||||
|
||||
__version__ = '2019.07.02'
|
||||
__version__ = '2019.07.30'
|
||||
|
||||
Loading…
Reference in New Issue
Block a user